Details
-
Type:
Story
-
Status: To Do
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: pipe_tasks
-
Labels:
-
Team:Data Release Production
-
Urgent?:No
Description
As Tim Jenness pointed out in DM-35752
it looks like the changes that were made to SkyCorrectionTask to handle missing datasets won't work once we fix the pipe_base problem. The discussion on this ticket indicates that the logic for handling absent datasets in the loop is missing
Error:
ERROR 2022-08-05T16:35:20.476-07:00 lsst.ctrl.mpexec.singleQuantumExecutor (skyCorr:{instrument: 'HSC', visit: 27130, ...})(singleQuantumExecutor.py:220) - Execution of task 'skyCorr' on quantum {instrument: 'HSC', visit: 27130, ...} failed. Exception AttributeError: 'NoneType' object has no attribute 'getMetadata' |
ERROR 2022-08-05T16:35:20.531-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:500) - Task <TaskDef(lsst.pipe.tasks.skyCorrection.SkyCorrectionTask, label=skyCorr) dataId={instrument: 'HSC', visit: 27130, ...}> failed; processing will continue for remaining tasks. |
Traceback (most recent call last):
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 470, in _executeQuantaInProcess |
self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 140, in execute |
result = self._execute(taskDef, quantum, butler)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 218, in _execute |
self.runQuantum(runTask, quantum, taskDef, butler)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 582, in runQuantum |
task.runQuantum(butlerQC, inputRefs, outputRefs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 281, in runQuantum |
outputs = self.run(**inputs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 406, in run |
skyExp = self.sky.exposureToBackground(skyCalib)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/background.py", line 162, in exposureToBackground |
header = bgExp.getMetadata()
|
AttributeError: 'NoneType' object has no attribute 'getMetadata' |
INFO 2022-08-05T16:35:21.060-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:516) - Executed 0 quanta successfully, 1 failed and 0 remain out of total 1 quanta. |
ERROR 2022-08-05T16:35:21.061-07:00 lsst.daf.butler.cli.utils ()(utils.py:1049) - Caught an exception, details are in traceback: |
Traceback (most recent call last):
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 131, in run |
script.run(qgraphObj=qgraph, **kwargs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run |
f.runPipeline(qgraphObj, taskFactory, args)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 751, in runPipeline |
executor.execute(graph, butler)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 401, in execute |
self._executeQuantaInProcess(graph, butler, self.report)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 526, in _executeQuantaInProcess |
raise MPGraphExecutorError("One or more tasks failed during execution.") |
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution.
|
To reproduce at USDF:
pipetask --long-log --log-level=VERBOSE run -b /sdf/group/rubin/repo/main_20220411/butler.yaml -i HSC/runs/RC2/w_2022_28/PREOPS-1225 --output u/hchiang2/w_2022_32/test -p ${DRP_PIPE_DIR}/pipelines/HSC/DRP-RC2.yaml#skyCorr -d "exposure = 27130" |
This is failing for the HSC-RC2 run step2 at USDF.
