Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10268

Butler cannot read a repo using the realpath when it was created with a link

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: daf_persistence
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Sprint:
      DB_S17_4, DB_S17_5
    • Team:
      Data Access and Database

      Description

      With the recent Butler changes (likely when _parent link was removed), Butler can no longer read an output repo if the repo path was from a symlink.
      For example, on lsst-dev, I have a symlink
      /datasets/hsc/repo/rerun/private/hchiang2/ -> /scratch/hchiang2/hscRerun/
      (therefore /datasets/hsc/repo/rerun/private/hchiang2/xxx ==
      /scratch/hchiang2/hscRerun/xxx)
      so I processed some hsc data and the outputs went to my scratch space
      (e.g. processCcd.py /datasets/hsc/repo --id visit=18194 ccd=9 --rerun private/hchiang2/xxx)

      Then I tried to read the output repo. I can only read it through dafPersist.Butler("/datasets/hsc/repo/rerun/private/hchiang2/xxx/"), but not dafPersist.Butler("/scratch/hchiang2/hscRerun/xxx/"). The latter had errors finding its parent:

      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-6-gf146911/python/lsst/daf/persistence/butler.py", line 353, in __init__
          self._repos._buildLookupLists(inputs, outputs)
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-6-gf146911/python/lsst/daf/persistence/butler.py", line 219, in _buildLookupLists
          addRepoDataToLists(repoData, 'out')
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-6-gf146911/python/lsst/daf/persistence/butler.py", line 211, in addRepoDataToLists
          addRepoDataToLists(self.byRepoRoot[parent], addParentAs)
      KeyError: '/'
      

      Above was with the w_2017_14 stack. The repositoryCfg.yaml file in the repo has
      _parents: [../../../..]

        Attachments

          Activity

            People

            • Assignee:
              npease Nate Pease
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Kenny Lo
              Watchers:
              Fritz Mueller, Hsin-Fang Chiang, Kenny Lo, Kian-Tat Lim, Nate Pease, Robert Lupton
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel