Andy Salnikov added a comment - 01/Aug/22 5:16 PM Tim Jenness , it should be ready for review. The change is trivial, and I added a unit test. Jenkins looks happy. PR: https://github.com/lsst/pipe_base/pull/269

Tim Jenness added a comment - 01/Aug/22 6:25 PM Looks great.

Hsin-Fang Chiang added a comment - 06/Aug/22 4:16 AM With w_2022_32, the same pipetask command still fails. The error message is slightly different: ERROR 2022 - 08 -05T16: 35 : 20.476 - 07 : 00 lsst.ctrl.mpexec.singleQuantumExecutor (skyCorr:{instrument: 'HSC' , visit: 27130 , ...})(singleQuantumExecutor.py: 220 ) - Execution of task 'skyCorr' on quantum {instrument: 'HSC' , visit: 27130 , ...} failed. Exception AttributeError: 'NoneType' object has no attribute 'getMetadata' ERROR 2022 - 08 -05T16: 35 : 20.531 - 07 : 00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py: 500 ) - Task <TaskDef(lsst.pipe.tasks.skyCorrection.SkyCorrectionTask, label=skyCorr) dataId={instrument: 'HSC' , visit: 27130 , ...}> failed; processing will continue for remaining tasks. Traceback (most recent call last): File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py" , line 470 , in _executeQuantaInProcess self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py" , line 140 , in execute result = self._execute(taskDef, quantum, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py" , line 218 , in _execute self.runQuantum(runTask, quantum, taskDef, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/singleQuantumExecutor.py" , line 582 , in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py" , line 281 , in runQuantum outputs = self.run(**inputs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/skyCorrection.py" , line 406 , in run skyExp = self.sky.exposureToBackground(skyCalib) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_tasks/g57dfbfe8b2+08e3ebb819/python/lsst/pipe/tasks/background.py" , line 162 , in exposureToBackground header = bgExp.getMetadata() AttributeError: 'NoneType' object has no attribute 'getMetadata' INFO 2022 - 08 -05T16: 35 : 21.060 - 07 : 00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py: 516 ) - Executed 0 quanta successfully, 1 failed and 0 remain out of total 1 quanta. ERROR 2022 - 08 -05T16: 35 : 21.061 - 07 : 00 lsst.daf.butler.cli.utils ()(utils.py: 1049 ) - Caught an exception, details are in traceback: Traceback (most recent call last): File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/cmd/commands.py" , line 131 , in run script.run(qgraphObj=qgraph, **kwargs) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cli/script/run.py" , line 187 , in run f.runPipeline(qgraphObj, taskFactory, args) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/cmdLineFwk.py" , line 751 , in runPipeline executor.execute(graph, butler) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py" , line 401 , in execute self._executeQuantaInProcess(graph, butler, self.report) File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2022_32/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g0eed036183+a288e28574/python/lsst/ctrl/mpexec/mpGraphExecutor.py" , line 526 , in _executeQuantaInProcess raise MPGraphExecutorError( "One or more tasks failed during execution." ) lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution. Or see the full log here https://storage.googleapis.com/drp-us-central1-logging/logs/SLAC_Rubin_Himem/PandaJob_24936833/payload.stderr

Hsin-Fang Chiang added a comment - 06/Aug/22 4:21 AM This can be used to reproduce at USDF pipetask -- long -log --log-level=VERBOSE run -b /sdf/group/rubin/repo/main_20220411/butler.yaml -i HSC/runs/RC2/w_2022_28/PREOPS- 1225 --output u/hchiang2/w_2022_32/test -p ${DRP_PIPE_DIR}/pipelines/HSC/DRP-RC2.yaml#skyCorr -d "exposure = 27130" Same as this ticket, it's just that the butler repo link might be moved anytime.

Tim Jenness added a comment - 06/Aug/22 4:30 AM Exactly as I said would happen. Look at some of the earlier comments where we note that even though there is an issue with the middleware, even if we were to fix that (which we did) there is a much bigger problem with sky correction task itself not working if a dataset is missing (which is what the None was indicating). You will probably need a specific ticket for SkyCorrectionTask (which is now in pipe_tasks) since the fix we did on this ticket was different.