Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35861

SkyCorrectionTask doesn't work with missing datasets

    XMLWordPrintable

    Details

    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As Tim Jenness pointed out in DM-35752

      it looks like the changes that were made to SkyCorrectionTask to handle missing datasets won't work once we fix the pipe_base problem. The discussion on this ticket indicates that the logic for handling absent datasets in the loop is missing

      Error:

      ERROR 2022-08-05T16:35:20.476-07:00 lsst.ctrl.mpexec.singleQuantumExecutor (skyCorr:{instrument: 'HSC', visit: 27130, ...})(singleQuantumExecutor.py:220) - Execution of task 'skyCorr' on quantum {instrument: 'HSC', visit: 27130, ...} failed. Exception AttributeError: 'NoneType' object has no attribute 'getMetadata'
      ERROR 2022-08-05T16:35:20.531-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:500) - Task <TaskDef(lsst.pipe.tasks.skyCorrection.SkyCorrectionTask, label=skyCorr) dataId={instrument: 'HSC', visit: 27130, ...}> failed; processing will continue for remaining tasks.
      Traceback (most recent call last):
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 470, in _executeQuantaInProcess
          self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 140, in execute
          result = self._execute(taskDef, quantum, butler)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 218, in _execute
          self.runQuantum(runTask, quantum, taskDef, butler)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 582, in runQuantum
          task.runQuantum(butlerQC, inputRefs, outputRefs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 281, in runQuantum
          outputs = self.run(**inputs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 406, in run
          skyExp = self.sky.exposureToBackground(skyCalib)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/background.py", line 162, in exposureToBackground
          header = bgExp.getMetadata()
      AttributeError: 'NoneType' object has no attribute 'getMetadata'
      INFO 2022-08-05T16:35:21.060-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:516) - Executed 0 quanta successfully, 1 failed and 0 remain out of total 1 quanta.
      ERROR 2022-08-05T16:35:21.061-07:00 lsst.daf.butler.cli.utils ()(utils.py:1049) - Caught an exception, details are in traceback:
      Traceback (most recent call last):
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 131, in run
          script.run(qgraphObj=qgraph, **kwargs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run
          f.runPipeline(qgraphObj, taskFactory, args)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 751, in runPipeline
          executor.execute(graph, butler)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 401, in execute
          self._executeQuantaInProcess(graph, butler, self.report)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 526, in _executeQuantaInProcess
          raise MPGraphExecutorError("One or more tasks failed during execution.")
      lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution.
      

      To reproduce at USDF:

      pipetask --long-log --log-level=VERBOSE run -b /sdf/group/rubin/repo/main_20220411/butler.yaml -i HSC/runs/RC2/w_2022_28/PREOPS-1225 --output u/hchiang2/w_2022_32/test -p ${DRP_PIPE_DIR}/pipelines/HSC/DRP-RC2.yaml#skyCorr  -d "exposure = 27130" 
      

      This is failing for the HSC-RC2 run step2 at USDF.

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            hchiang2 Hsin-Fang Chiang
            Watchers:
            Hsin-Fang Chiang, Jim Bosch, Tim Jenness, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:

                Jenkins

                No builds found.