Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14753

Butler with multiple inputs does not properly load registry

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:

      butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])

      One clue is that this prints the following INFO:

      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      		 
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      		 
      CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
      		 
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      		 
      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      		 
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      		 
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3

      And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

      Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
      I believe that on the first call to _setParentRegistry() this argument is `None` (not sure why), but on subsequent calls (after repoData.repo = Repository(repoData) is called) then it's okay.

      I checked what would happen if instead of kicking out instead it calls parentRepoData.repo = Repository(parentRepoData) and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14669 Run fgcmcal on HSC PDR1

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            erykoff Eli Rykoff created issue -
            erykoff Eli Rykoff made changes -
            Field Original Value New Value
            Epic Link DM-14455 [ 80721 ]
            erykoff Eli Rykoff made changes -
            Link This issue blocks DM-14669 [ DM-14669 ]
            erykoff Eli Rykoff made changes -
            Risk Score 0
            erykoff Eli Rykoff made changes -
            Description When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:
            {{
            butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
            }}
            One clue is that this prints the following INFO:
            {{
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            }}
            And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

            Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
            I believe that on the first call to {{_setParentRegistry()}} this argument is `None` (not sure why), but on subsequent calls (after {{repoData.repo = Repository(repoData)}} is called) then it's okay.

            I checked what would happen if instead of kicking out instead it calls {{parentRepoData.repo = Repository(parentRepoData)}} and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).             		When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:
            {code: python}
            butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
            {code}
            One clue is that this prints the following INFO:
            {code: python}
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            {code}
            And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

            Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
            I believe that on the first call to {{_setParentRegistry()}} this argument is `None` (not sure why), but on subsequent calls (after {{repoData.repo = Repository(repoData)}} is called) then it's okay.

            I checked what would happen if instead of kicking out instead it calls {{parentRepoData.repo = Repository(parentRepoData)}} and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Wondering if Nate Pease [X] thinks this is reasonable. I can put it on a branch...

            Show
            erykoff Eli Rykoff added a comment - Wondering if Nate Pease [X] thinks this is reasonable. I can put it on a branch...
            npease Nate Pease [X] (Inactive) made changes -
            Assignee Nate Pease [ npease ]
            npease Nate Pease [X] (Inactive) made changes -
            Story Points 1 2
            npease Nate Pease [X] (Inactive) made changes -
            Team Data Release Production [ 10301 ] Data Access and Database [ 10204 ]
            npease Nate Pease [X] (Inactive) made changes -
            Sprint AP S18-6 [ 686 ] AP S18-6, DB_F18_06 [ 686, 744 ]
            npease Nate Pease [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            npease Nate Pease [X] (Inactive) added a comment -

            Kian-Tat Lim can you please review?

            Show
            npease Nate Pease [X] (Inactive) added a comment - Kian-Tat Lim can you please review?
            npease Nate Pease [X] (Inactive) made changes -
            Reviewers Kian-Tat Lim [ ktl ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            I can't certify that it is correct or solves the issue (a test case for that would be nice, if it's possible to write one quickly), but the code looks reasonably clear.

            Show
            ktl Kian-Tat Lim added a comment - I can't certify that it is correct or solves the issue (a test case for that would be nice, if it's possible to write one quickly), but the code looks reasonably clear.
            ktl Kian-Tat Lim made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            I will at least confirm that it solves my problem above. But it would be worth making that into a test case, but I don't know how many parents of parents we need to emulate.

            Show
            erykoff Eli Rykoff added a comment - I will at least confirm that it solves my problem above. But it would be worth making that into a test case, but I don't know how many parents of parents we need to emulate.
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            And it works for my use case listed above!

            Show
            erykoff Eli Rykoff added a comment - And it works for my use case listed above!
            npease Nate Pease [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              npease Nate Pease [X] (Inactive)
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Eli Rykoff, Kian-Tat Lim, Nate Pease [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.