Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11333

Confusion in how Butler stores and compares calibRoot

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: daf_persistence
    • Labels:
    • Templates:
    • Team:
      Data Access and Database

      Description

      Using w_2017_28, we seem to have found a loophole in how butler compares the repository config.

      With HscMapper the default calib repo is CALIB and on lsst-dev we set a link

      /datasets/hsc/repo/CALIB -> /datasets/hsc/calib/20170105  
      

      We can do the following:

      $ processCcd.py /datasets/hsc/repo/ --output ~/a --id visit=903990 ccd=24   --calib /datasets/hsc/calib/20170105/ 
      $ processCcd.py /datasets/hsc/repo/ --output ~/a --id visit=903990 ccd=25   
      

      Both finish processing just fine. ~/a/repositoryCfg.yaml has

      _mapperArgs: {calibRoot: /datasets/hsc/calib/20170105}  
      

      However, if we swap the order of processing commands, and do the following:

      $  processCcd.py /datasets/hsc/repo/ --output ~/b --id visit=903990 ccd=24 
      $  processCcd.py /datasets/hsc/repo/ --output ~/b --id visit=903990 ccd=25 --calib /datasets/hsc/calib/20170105/ 
      

      ~/b/repositoryCfg.yaml has

      _mapperArgs: null
      

      and the second processing fails with

      Traceback (most recent call last):
        File "/software/lsstsw/stack/Linux64/pipe_tasks/13.0-44-g89df7c8/bin/processCcd.py", line 25, in <module>
          ProcessCcdTask.parseAndRun()
        File "/software/lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 509, in parseAndRun
          parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
        File "/software/lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/argumentParser.py", line 512, in parse_args
          namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs)
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py", line 527, in __init__
          self._getCfgs(repoDataList)
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py", line 797, in _getCfgs
          cfg, repoData.repoArgs))
      RuntimeError: The RepositoryArgs and RepositoryCfg must match for writable repositories, RepositoryCfg:RepositoryCfg(root='/home/hchiang2/b', mapper=<class 'lsst.obs.hsc.hscMapper.HscMapper'>, mapperArgs=None, parents=['../../../datasets/hsc/repo'], policy=None), RepositoryArgs:RepositoryArgs(root='/home/hchiang2/b', cfgRoot=None, mapper=None, mapperArgs={'calibRoot': '/datasets/hsc/calib/20170105'}, tags=set([]), mode='rw', policy=None)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Michelle Gower
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel