Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11217

Butler+CmdLineTask cannot output to a (non-repo) folder where some files already exist

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: daf_persistence
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Sprint:
      DB_S17_7
    • Team:
      Data Access and Database

      Description

      Samantha Thrush noticed that tasks fail if the output is a non-empty non-repo folder. To reproduce with w_2017_26:

      $ mkdir out
      $ echo 1234 > out/a
      $ processCcd.py /datasets/hsc/repo --id visit=444 ccd=4  --output out
      

      This gives errors:

      Traceback (most recent call last):
        File "/software/lsstsw/stack/Linux64/pipe_tasks/13.0-38-gf73ba12/bin/processCcd.py", line 25, in <module>
          ProcessCcdTask.parseAndRun()
        File "/software/lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/cmdLineTask.py", line 509, in parseAndRun
          parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
        File "/software/lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/argumentParser.py", line 512, in parse_args
          namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs)
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-20-g8cd6840/python/lsst/daf/persistence/butler.py", line 531, in __init__
          self._setAndVerifyParentsLists(repoDataList)
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-20-g8cd6840/python/lsst/daf/persistence/butler.py", line 905, in _setAndVerifyParentsLists
          repoData.cfg.extendParents(parents)
        File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-20-g8cd6840/python/lsst/daf/persistence/repositoryCfg.py", line 167, in extendParents
          if all(x == y for (x, y) in zip(self._parents, newParents)):
      TypeError: zip argument #1 must support iteration
      

      If the folder is empty (i.e. skipping $ echo 1234 > out/a in the example) then the task runs just fine.

        Attachments

          Activity

            People

            • Assignee:
              npease Nate Pease
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Hsin-Fang Chiang, Kian-Tat Lim, Nate Pease, Samantha Thrush
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel