Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11284

Butler writes many copies of identical entries _parents to repositoryCfg.yaml

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Templates:
    • Story Points:
      4
    • Sprint:
      DB_S17_7
    • Team:
      Data Access and Database

      Description

      I just noticed that one of the calibration scripts is loading the same Posix calibration registry around 40 times (which isn't cheap).

      The problem seems to be that rerun/rhl/tmp/repositoryCfg.yaml has many copies of the parent in its _parents list:

      !RepositoryCfg_v1
      _mapper: !!python/name:lsst.obs.pfs.pfsMapper.PfsMapper ''
      _mapperArgs: {calibRoot: /datasets/PFS/LAM/XCALIB}
      _parents: [../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM]
      _policy: null
      _root: null
      dirty: true
      

      While poking at this, I noted that the

      python/lsst/daf/persistence/butler.py", line 909, in _setAndVerifyParentsLists
          parents, repoData.cfg.parents, e))
      RuntimeError: Inputs of this Butler: ...
      

      error message doesn't give the paths to the butlers in question – if it did, it'd make the problem much easier to debug.

        Attachments

          Activity

            People

            • Assignee:
              npease Nate Pease
              Reporter:
              rhl Robert Lupton
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Kian-Tat Lim, Nate Pease, Paul Price, Robert Lupton
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel