Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14753

Butler with multiple inputs does not properly load registry

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:

      butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
      

      One clue is that this prints the following INFO:

      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      

      And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

      Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
      I believe that on the first call to _setParentRegistry() this argument is `None` (not sure why), but on subsequent calls (after repoData.repo = Repository(repoData) is called) then it's okay.

      I checked what would happen if instead of kicking out instead it calls parentRepoData.repo = Repository(parentRepoData) and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).

      Attachments

        Issue Links

          Activity

            erykoff Eli Rykoff added a comment -

            Wondering if npease thinks this is reasonable. I can put it on a branch...

            erykoff Eli Rykoff added a comment - Wondering if npease thinks this is reasonable. I can put it on a branch...

            ktl can you please review?

            npease Nate Pease [X] (Inactive) added a comment - ktl can you please review?

            I can't certify that it is correct or solves the issue (a test case for that would be nice, if it's possible to write one quickly), but the code looks reasonably clear.

            ktl Kian-Tat Lim added a comment - I can't certify that it is correct or solves the issue (a test case for that would be nice, if it's possible to write one quickly), but the code looks reasonably clear.
            erykoff Eli Rykoff added a comment -

            I will at least confirm that it solves my problem above. But it would be worth making that into a test case, but I don't know how many parents of parents we need to emulate.

            erykoff Eli Rykoff added a comment - I will at least confirm that it solves my problem above. But it would be worth making that into a test case, but I don't know how many parents of parents we need to emulate.
            erykoff Eli Rykoff added a comment -

            And it works for my use case listed above!

            erykoff Eli Rykoff added a comment - And it works for my use case listed above!

            People

              npease Nate Pease [X] (Inactive)
              erykoff Eli Rykoff
              Kian-Tat Lim
              Eli Rykoff, Kian-Tat Lim, Nate Pease [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.