Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14753

Butler with multiple inputs does not properly load registry

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:

      butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
      

      One clue is that this prints the following INFO:

      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      

      And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

      Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
      I believe that on the first call to _setParentRegistry() this argument is `None` (not sure why), but on subsequent calls (after repoData.repo = Repository(repoData) is called) then it's okay.

      I checked what would happen if instead of kicking out instead it calls parentRepoData.repo = Repository(parentRepoData) and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).

        Attachments

          Issue Links

            Activity

            erykoff Eli Rykoff created issue -
            erykoff Eli Rykoff made changes -
            Field Original Value New Value
            Epic Link DM-14455 [ 80721 ]
            erykoff Eli Rykoff made changes -
            Link This issue blocks DM-14669 [ DM-14669 ]
            erykoff Eli Rykoff made changes -
            Risk Score 0
            erykoff Eli Rykoff made changes -
            Description When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:
            {{
            butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
            }}
            One clue is that this prints the following INFO:
            {{
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            }}
            And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

            Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
            I believe that on the first call to {{_setParentRegistry()}} this argument is `None` (not sure why), but on subsequent calls (after {{repoData.repo = Repository(repoData)}} is called) then it's okay.

            I checked what would happen if instead of kicking out instead it calls {{parentRepoData.repo = Repository(parentRepoData)}} and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).
            When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:
            {code: python}
            butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
            {code}
            One clue is that this prints the following INFO:
            {code: python}
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            {code}
            And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

            Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
            I believe that on the first call to {{_setParentRegistry()}} this argument is `None` (not sure why), but on subsequent calls (after {{repoData.repo = Repository(repoData)}} is called) then it's okay.

            I checked what would happen if instead of kicking out instead it calls {{parentRepoData.repo = Repository(parentRepoData)}} and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).
            npease Nate Pease [X] (Inactive) made changes -
            Assignee Nate Pease [ npease ]
            npease Nate Pease [X] (Inactive) made changes -
            Story Points 1 2
            npease Nate Pease [X] (Inactive) made changes -
            Team Data Release Production [ 10301 ] Data Access and Database [ 10204 ]
            npease Nate Pease [X] (Inactive) made changes -
            Sprint AP S18-6 [ 686 ] AP S18-6, DB_F18_06 [ 686, 744 ]
            npease Nate Pease [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            npease Nate Pease [X] (Inactive) made changes -
            Reviewers Kian-Tat Lim [ ktl ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            ktl Kian-Tat Lim made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            npease Nate Pease [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              npease Nate Pease [X] (Inactive)
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Eli Rykoff, Kian-Tat Lim, Nate Pease [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.