Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9931

Add dependency of "refcat" on "preSfm" in ci_hsc

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None

      Description

      ci_hsc fails in my lsst-dev/lsstvc/slurm env (using 1 node) although it passes on Jenkins.
      The problem arose when preSfm ran before refcat.

      I think this is because preSfm, effectively processCcd.py DATA --rerun ci_hsc --doraise, needs the ref_cats link already in the repo. Running the command in a repo without the ref_cats gives this error:

      root INFO: Running: /home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/bin/processCcd.py /home/hchiang2/ciHscTest/asMaster/ci_hsc/DATA --rerun test2 --doraise
      Traceback (most recent call last):
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/bin/processCcd.py", line 25, in <module>
          ProcessCcdTask.parseAndRun()
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 482, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 202, in run
          if self.precall(parsedCmd):
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 300, in precall
          task = self.makeTask(parsedCmd=parsedCmd)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 391, in makeTask
          return self.TaskClass(config=self.config, log=self.log, butler=butler)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/python/lsst/pipe/tasks/processCcd.py", line 160, in __init__
          astromRefObjLoader=astromRefObjLoader, photoRefObjLoader=photoRefObjLoader)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/task.py", line 237, in makeSubtask
          subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        File "/home/hchiang2/lsstsw/stack/Linux64/pex_config/13.0-1-g41367f3+1/python/lsst/pex/config/configurableField.py", line 83, in apply
          return self.target(*args, config=self.value, **kw)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/python/lsst/pipe/tasks/calibrate.py", line 322, in __init__
          self.makeSubtask('astromRefObjLoader', butler=butler)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/task.py", line 237, in makeSubtask
          subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        File "/home/hchiang2/lsstsw/stack/Linux64/pex_config/13.0-1-g41367f3+1/python/lsst/pex/config/configurableField.py", line 83, in apply
          return self.target(*args, config=self.value, **kw)
        File "/home/hchiang2/lsstsw/stack/Linux64/meas_algorithms/13.0-6-g6f8a914+2/python/lsst/meas/algorithms/loadIndexedReferenceObjects.py", line 50, in __init__
          dataset_config = butler.get("ref_cat_config", name=self.config.ref_dataset_name, immediate=True)
        File "/home/hchiang2/lsstsw/stack/Linux64/daf_persistence/13.0-5-g87674b4/python/lsst/daf/persistence/butler.py", line 945, in get
          raise NoResults("No locations for get:", datasetType, dataId)
      lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'ps1_pv3_3pi_20170110'}, tag=set([]))
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Paul Price
                Watchers:
                Hsin-Fang Chiang, John Swinbank, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel