Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9931

Add dependency of "refcat" on "preSfm" in ci_hsc

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None

      Description

      ci_hsc fails in my lsst-dev/lsstvc/slurm env (using 1 node) although it passes on Jenkins.
      The problem arose when preSfm ran before refcat.

      I think this is because preSfm, effectively processCcd.py DATA --rerun ci_hsc --doraise, needs the ref_cats link already in the repo. Running the command in a repo without the ref_cats gives this error:

      root INFO: Running: /home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/bin/processCcd.py /home/hchiang2/ciHscTest/asMaster/ci_hsc/DATA --rerun test2 --doraise
      Traceback (most recent call last):
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/bin/processCcd.py", line 25, in <module>
          ProcessCcdTask.parseAndRun()
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 482, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 202, in run
          if self.precall(parsedCmd):
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 300, in precall
          task = self.makeTask(parsedCmd=parsedCmd)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/cmdLineTask.py", line 391, in makeTask
          return self.TaskClass(config=self.config, log=self.log, butler=butler)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/python/lsst/pipe/tasks/processCcd.py", line 160, in __init__
          astromRefObjLoader=astromRefObjLoader, photoRefObjLoader=photoRefObjLoader)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/task.py", line 237, in makeSubtask
          subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        File "/home/hchiang2/lsstsw/stack/Linux64/pex_config/13.0-1-g41367f3+1/python/lsst/pex/config/configurableField.py", line 83, in apply
          return self.target(*args, config=self.value, **kw)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_tasks/13.0-8-g148fcaf+1/python/lsst/pipe/tasks/calibrate.py", line 322, in __init__
          self.makeSubtask('astromRefObjLoader', butler=butler)
        File "/home/hchiang2/lsstsw/stack/Linux64/pipe_base/13.0-1-g92030ba+10/python/lsst/pipe/base/task.py", line 237, in makeSubtask
          subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        File "/home/hchiang2/lsstsw/stack/Linux64/pex_config/13.0-1-g41367f3+1/python/lsst/pex/config/configurableField.py", line 83, in apply
          return self.target(*args, config=self.value, **kw)
        File "/home/hchiang2/lsstsw/stack/Linux64/meas_algorithms/13.0-6-g6f8a914+2/python/lsst/meas/algorithms/loadIndexedReferenceObjects.py", line 50, in __init__
          dataset_config = butler.get("ref_cat_config", name=self.config.ref_dataset_name, immediate=True)
        File "/home/hchiang2/lsstsw/stack/Linux64/daf_persistence/13.0-5-g87674b4/python/lsst/daf/persistence/butler.py", line 945, in get
          raise NoResults("No locations for get:", datasetType, dataId)
      lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'ps1_pv3_3pi_20170110'}, tag=set([]))
      
      

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            So now I'm confused. I thought Paul Price agreed with me that the solution proposed here probably isn't right, but then he went ahead and marked it "reviewed" without comment. This seems wrong...!

            Show
            swinbank John Swinbank added a comment - So now I'm confused. I thought Paul Price agreed with me that the solution proposed here probably isn't right, but then he went ahead and marked it "reviewed" without comment. This seems wrong...!
            Hide
            price Paul Price added a comment -

            The implementation has changed to what we suggested.

            Show
            price Paul Price added a comment - The implementation has changed to what we suggested.
            Hide
            swinbank John Swinbank added a comment -

            Show
            swinbank John Swinbank added a comment -
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thank you both very much for pointing me to the correct way of using scons!

            I pushed the new fix but missed to comment here. Sorry for the confusion. Now test-running on lsstvc and Jenkins before merging it. Also fixed the descriptions.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thank you both very much for pointing me to the correct way of using scons! I pushed the new fix but missed to comment here. Sorry for the confusion. Now test-running on lsstvc and Jenkins before merging it. Also fixed the descriptions.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Merged after passing Jenkins and the slurm env that I saw this originally.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Merged after passing Jenkins and the slurm env that I saw this originally.

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Paul Price
                Watchers:
                Hsin-Fang Chiang, John Swinbank, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel