[DM-35861] SkyCorrectionTask doesn't work with missing datasets - Jira

XML

Word

Printable

Details

Type: Story
Status: To Do
Resolution: Unresolved
Fix Version/s: None
Component/s: pipe_tasks
Labels:
- SciencePipelines

Team:
Data Release Production
Urgent?:
No

Description

As Tim Jenness pointed out in ~~DM-35752~~

it looks like the changes that were made to SkyCorrectionTask to handle missing datasets won't work once we fix the pipe_base problem. The discussion on this ticket indicates that the logic for handling absent datasets in the loop is missing

Error:

ERROR 2022-08-05T16:35:20.476-07:00 lsst.ctrl.mpexec.singleQuantumExecutor (skyCorr:{instrument: 'HSC', visit: 27130, ...})(singleQuantumExecutor.py:220) - Execution of task 'skyCorr' on quantum {instrument: 'HSC', visit: 27130, ...} failed. Exception AttributeError: 'NoneType' object has no attribute 'getMetadata'

ERROR 2022-08-05T16:35:20.531-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:500) - Task <TaskDef(lsst.pipe.tasks.skyCorrection.SkyCorrectionTask, label=skyCorr) dataId={instrument: 'HSC', visit: 27130, ...}> failed; processing will continue for remaining tasks.

Traceback (most recent call last):

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 470, in _executeQuantaInProcess

    self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 140, in execute

    result = self._execute(taskDef, quantum, butler)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 218, in _execute

    self.runQuantum(runTask, quantum, taskDef, butler)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 582, in runQuantum

    task.runQuantum(butlerQC, inputRefs, outputRefs)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 281, in runQuantum

    outputs = self.run(**inputs)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 406, in run

    skyExp = self.sky.exposureToBackground(skyCalib)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/background.py", line 162, in exposureToBackground

    header = bgExp.getMetadata()

AttributeError: 'NoneType' object has no attribute 'getMetadata'

INFO 2022-08-05T16:35:21.060-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:516) - Executed 0 quanta successfully, 1 failed and 0 remain out of total 1 quanta.

ERROR 2022-08-05T16:35:21.061-07:00 lsst.daf.butler.cli.utils ()(utils.py:1049) - Caught an exception, details are in traceback:

Traceback (most recent call last):

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 131, in run

    script.run(qgraphObj=qgraph, **kwargs)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run

    f.runPipeline(qgraphObj, taskFactory, args)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 751, in runPipeline

    executor.execute(graph, butler)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 401, in execute

    self._executeQuantaInProcess(graph, butler, self.report)

  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 526, in _executeQuantaInProcess

    raise MPGraphExecutorError("One or more tasks failed during execution.")

lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution.

To reproduce at USDF:

pipetask --long-log --log-level=VERBOSE run -b /sdf/group/rubin/repo/main_20220411/butler.yaml -i HSC/runs/RC2/w_2022_28/PREOPS-1225 --output u/hchiang2/w_2022_32/test -p ${DRP_PIPE_DIR}/pipelines/HSC/DRP-RC2.yaml#skyCorr  -d "exposure = 27130"

This is failing for the HSC-RC2 run step2 at USDF.

Attachments

Activity

Ascending order - Click to sort in descending order

No builds found.

Hsin-Fang Chiang created issue - 07/Aug/22 7:16 PM

Hsin-Fang Chiang made changes - 07/Aug/22 7:20 PM

Field	Original Value	New Value
Description	As [~tjenness] pointed out in ~~DM-35752~~ {noformat} it looks like the changes that were made to SkyCorrectionTask to handle missing datasets won't work once we fix the pipe_base problem. The discussion on this ticket indicates that the logic for handling absent datasets in the loop is missing {noformat} Error: {code:java} ERROR 2022-08-05T16:35:20.476-07:00 lsst.ctrl.mpexec.singleQuantumExecutor (skyCorr:{instrument: 'HSC', visit: 27130, ...})(singleQuantumExecutor.py:220) - Execution of task 'skyCorr' on quantum {instrument: 'HSC', visit: 27130, ...} failed. Exception AttributeError: 'NoneType' object has no attribute 'getMetadata' ERROR 2022-08-05T16:35:20.531-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:500) - Task <TaskDef(lsst.pipe.tasks.skyCorrection.SkyCorrectionTask, label=skyCorr) dataId={instrument: 'HSC', visit: 27130, ...}> failed; processing will continue for remaining tasks. Traceback (most recent call last): File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 470, in _executeQuantaInProcess self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 140, in execute result = self._execute(taskDef, quantum, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 218, in _execute self.runQuantum(runTask, quantum, taskDef, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 582, in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 281, in runQuantum outputs = self.run(inputs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 406, in run skyExp = self.sky.exposureToBackground(skyCalib) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/background.py", line 162, in exposureToBackground header = bgExp.getMetadata() AttributeError: 'NoneType' object has no attribute 'getMetadata' INFO 2022-08-05T16:35:21.060-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:516) - Executed 0 quanta successfully, 1 failed and 0 remain out of total 1 quanta. ERROR 2022-08-05T16:35:21.061-07:00 lsst.daf.butler.cli.utils ()(utils.py:1049) - Caught an exception, details are in traceback: Traceback (most recent call last): File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 131, in run script.run(qgraphObj=qgraph, kwargs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run f.runPipeline(qgraphObj, taskFactory, args) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 751, in runPipeline executor.execute(graph, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 401, in execute self._executeQuantaInProcess(graph, butler, self.report) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 526, in _executeQuantaInProcess raise MPGraphExecutorError("One or more tasks failed during execution.") lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution. {code} To reproduce at USDF: {code:java} pipetask --long-log --log-level=VERBOSE run -b /sdf/group/rubin/repo/main_20220411/butler.yaml -i HSC/runs/RC2/w_2022_28/PREOPS-1225 --output u/hchiang2/w_2022_32/test -p ${DRP_PIPE_DIR}/pipelines/HSC/DRP-RC2.yaml#skyCorr -d "exposure = 27130" {code} This is failing for the HSC-RC2 run step2 at USDF.	As [~tjenness] pointed out in ~~DM-35752~~ bq. it looks like the changes that were made to SkyCorrectionTask to handle missing datasets won't work once we fix the pipe_base problem. The discussion on this ticket indicates that the logic for handling absent datasets in the loop is missing Error: {code:java} ERROR 2022-08-05T16:35:20.476-07:00 lsst.ctrl.mpexec.singleQuantumExecutor (skyCorr:{instrument: 'HSC', visit: 27130, ...})(singleQuantumExecutor.py:220) - Execution of task 'skyCorr' on quantum {instrument: 'HSC', visit: 27130, ...} failed. Exception AttributeError: 'NoneType' object has no attribute 'getMetadata' ERROR 2022-08-05T16:35:20.531-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:500) - Task <TaskDef(lsst.pipe.tasks.skyCorrection.SkyCorrectionTask, label=skyCorr) dataId={instrument: 'HSC', visit: 27130, ...}> failed; processing will continue for remaining tasks. Traceback (most recent call last): File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 470, in _executeQuantaInProcess self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 140, in execute result = self._execute(taskDef, quantum, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 218, in _execute self.runQuantum(runTask, quantum, taskDef, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 582, in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 281, in runQuantum outputs = self.run(inputs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py", line 406, in run skyExp = self.sky.exposureToBackground(skyCalib) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/background.py", line 162, in exposureToBackground header = bgExp.getMetadata() AttributeError: 'NoneType' object has no attribute 'getMetadata' INFO 2022-08-05T16:35:21.060-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:516) - Executed 0 quanta successfully, 1 failed and 0 remain out of total 1 quanta. ERROR 2022-08-05T16:35:21.061-07:00 lsst.daf.butler.cli.utils ()(utils.py:1049) - Caught an exception, details are in traceback: Traceback (most recent call last): File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 131, in run script.run(qgraphObj=qgraph, kwargs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run f.runPipeline(qgraphObj, taskFactory, args) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 751, in runPipeline executor.execute(graph, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 401, in execute self._executeQuantaInProcess(graph, butler, self.report) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 526, in _executeQuantaInProcess raise MPGraphExecutorError("One or more tasks failed during execution.") lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution. {code} To reproduce at USDF: {code:java} pipetask --long-log --log-level=VERBOSE run -b /sdf/group/rubin/repo/main_20220411/butler.yaml -i HSC/runs/RC2/w_2022_28/PREOPS-1225 --output u/hchiang2/w_2022_32/test -p ${DRP_PIPE_DIR}/pipelines/HSC/DRP-RC2.yaml#skyCorr -d "exposure = 27130" {code} This is failing for the HSC-RC2 run step2 at USDF.

Hide

Permalink

Hsin-Fang Chiang added a comment - 14/Aug/22 4:15 AM

Alternatively the w_2022_32's reprocessing outputs on USDF can be used to reproduce the problem too; the collection name is HSC/runs/RC2/w_2022_32/PREOPS-1225

Show

Hsin-Fang Chiang added a comment - 14/Aug/22 4:15 AM Alternatively the w_2022_32's reprocessing outputs on USDF can be used to reproduce the problem too; the collection name is HSC/runs/RC2/w_2022_32/PREOPS-1225

Hide

Permalink

Jim Bosch added a comment - 15/Aug/22 8:06 PM

It looks like the ultimate cause of this problem was a butler import file not being executed at USDF after file transfers, leading to some sky calibration datasets not being present. I think there's still work to do in SkyCorrectionTask, but it's probably to raise a sensible error message when a sky frame is not present for an input data ID rather than to proceed without it.

Show

Jim Bosch added a comment - 15/Aug/22 8:06 PM It looks like the ultimate cause of this problem was a butler import file not being executed at USDF after file transfers, leading to some sky calibration datasets not being present. I think there's still work to do in SkyCorrectionTask, but it's probably to raise a sensible error message when a sky frame is not present for an input data ID rather than to proceed without it.

Tim Jenness made changes - 15/Aug/22 9:08 PM

Component/s

pipe_tasks [ 10726 ]

Hide

Permalink

Hsin-Fang Chiang added a comment - 16/Aug/22 9:42 PM

Thanks to Jim Bosch for fixing the repo. Now SkyCorrectionTask runs fine.

Show

Hsin-Fang Chiang added a comment - 16/Aug/22 9:42 PM Thanks to Jim Bosch for fixing the repo. Now SkyCorrectionTask runs fine.

People

Assignee:: Unassigned

Reporter:: Hsin-Fang Chiang

Watchers:: Hsin-Fang Chiang, Jim Bosch, Tim Jenness, Yusra AlSayyad

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 07/Aug/22 7:16 PM

Updated:: 16/Aug/22 9:42 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Activity

People

Dates

Jenkins