Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14753

Butler with multiple inputs does not properly load registry

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:

      butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
      

      One clue is that this prints the following INFO:

      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      

      And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

      Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
      I believe that on the first call to _setParentRegistry() this argument is `None` (not sure why), but on subsequent calls (after repoData.repo = Repository(repoData) is called) then it's okay.

      I checked what would happen if instead of kicking out instead it calls parentRepoData.repo = Repository(parentRepoData) and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              npease Nate Pease [X] (Inactive)
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Eli Rykoff, Kian-Tat Lim, Nate Pease [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.