Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14753

Butler with multiple inputs does not properly load registry

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:

      butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
      

      One clue is that this prints the following INFO:

      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      

      And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)

      Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
      I believe that on the first call to _setParentRegistry() this argument is `None` (not sure why), but on subsequent calls (after repoData.repo = Repository(repoData) is called) then it's okay.

      I checked what would happen if instead of kicking out instead it calls parentRepoData.repo = Repository(parentRepoData) and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            Wondering if Nate Pease [X] thinks this is reasonable. I can put it on a branch...

            Show
            erykoff Eli Rykoff added a comment - Wondering if Nate Pease [X] thinks this is reasonable. I can put it on a branch...
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            Kian-Tat Lim can you please review?

            Show
            npease Nate Pease [X] (Inactive) added a comment - Kian-Tat Lim can you please review?
            Hide
            ktl Kian-Tat Lim added a comment -

            I can't certify that it is correct or solves the issue (a test case for that would be nice, if it's possible to write one quickly), but the code looks reasonably clear.

            Show
            ktl Kian-Tat Lim added a comment - I can't certify that it is correct or solves the issue (a test case for that would be nice, if it's possible to write one quickly), but the code looks reasonably clear.
            Hide
            erykoff Eli Rykoff added a comment -

            I will at least confirm that it solves my problem above. But it would be worth making that into a test case, but I don't know how many parents of parents we need to emulate.

            Show
            erykoff Eli Rykoff added a comment - I will at least confirm that it solves my problem above. But it would be worth making that into a test case, but I don't know how many parents of parents we need to emulate.
            Hide
            erykoff Eli Rykoff added a comment -

            And it works for my use case listed above!

            Show
            erykoff Eli Rykoff added a comment - And it works for my use case listed above!

              People

              Assignee:
              npease Nate Pease [X] (Inactive)
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Eli Rykoff, Kian-Tat Lim, Nate Pease [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.