Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30838

Disable bright-object-mask-based catalog flags for ImSim/DC2

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst, pipe_tasks
    • Labels:
      None
    • Story Points:
      1
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      DM-30649 correctly disabled bright object masks in AssembleCoadd for obs_lsst (we don't have any such masks for ImSim, and as they're an HSC special that's going away, we don't plan to make them). But the PixelFlags plugin is still configured to do something with them, and that needs to be fixed.

        Attachments

          Issue Links

            Activity

            Hide
            lskelvin Lee Kelvin added a comment - - edited

            Thanks for setting up the ticket Jim. In summary, I've been trying to reprocess DC2 data using w25. All of singleFrame succeeded without issue, and the first 5 tasks of multiVisit also passed without issue (makeWarp, assembleCoadd, detection, mergeDetections and deblend). Unfortunately, measure fails with the fatal algorithm error Invalid mask plane name: BRIGHT_OBJECT. I've attached the full traceback to this ticket: imsim_measure_error.log.

            My run command for this task was:

            pipetask --long-log run --register-dataset-types \
            -b /repo/dc2 --instrument lsst.obs.lsst.LsstCamImSim \
            -i u/lskelvin/DM-30820/default \
            --output-run u/lskelvin/DM-30820/testing \
            -p $OBS_LSST_DIR/pipelines/imsim/DRP.yaml#measure \
            -d "instrument='LSSTCam-imSim' AND skymap='DC2' AND tract IN (3828, 3263) AND patch IN (24, 21)"
            

            Following discussion on Slack, it appears that this issue may be linked to DM-30649, as indicated above. I attempted to turn back on the missing config option using assembleCoadd:doMaskBrightObjects=True, but this results in:

            FileNotFoundError: Not enough datasets (0) found for non-optional connection assembleCoadd.brightObjectMask (brightObjectMask) with minimum=1 for quantum data ID {band: 'r', skymap: 'DC2', tract: 3828, patch: 24}.
            

            As a final test, I rolled back to w24, and re-ran all of multiVisit. This run succeeded, so I suspect recent changes are the culprit here.

            Show
            lskelvin Lee Kelvin added a comment - - edited Thanks for setting up the ticket Jim. In summary, I've been trying to reprocess DC2 data using w25. All of singleFrame succeeded without issue, and the first 5 tasks of multiVisit also passed without issue ( makeWarp , assembleCoadd , detection , mergeDetections and deblend ). Unfortunately, measure fails with the fatal algorithm error Invalid mask plane name: BRIGHT_OBJECT . I've attached the full traceback to this ticket: imsim_measure_error.log . My run command for this task was: pipetask --long-log run --register-dataset-types \ -b /repo/dc2 --instrument lsst.obs.lsst.LsstCamImSim \ -i u/lskelvin/DM-30820/default \ --output-run u/lskelvin/DM-30820/testing \ -p $OBS_LSST_DIR/pipelines/imsim/DRP.yaml#measure \ -d "instrument='LSSTCam-imSim' AND skymap='DC2' AND tract IN (3828, 3263) AND patch IN (24, 21)" Following discussion on Slack, it appears that this issue may be linked to DM-30649 , as indicated above. I attempted to turn back on the missing config option using assembleCoadd:doMaskBrightObjects=True , but this results in: FileNotFoundError: Not enough datasets (0) found for non-optional connection assembleCoadd.brightObjectMask (brightObjectMask) with minimum=1 for quantum data ID {band: 'r', skymap: 'DC2', tract: 3828, patch: 24}. As a final test, I rolled back to w24, and re-ran all of multiVisit . This run succeeded, so I suspect recent changes are the culprit here.
            Hide
            jbosch Jim Bosch added a comment -

            This has passed ci_imsim in Jenkins (vanilla + ci_hsc Jenkins is still running), so I've now got some confidence that this at least does no harm, and Lee Kelvin's tests seem to show it helps. Let's get it reviewed now while I investigate why ci_imsim wasn't broken before, since there's a good chance that investigation won't involve any code changes.

            PRs are https://github.com/lsst/obs_lsst/pull/319 and https://github.com/lsst/pipe_tasks/pull/538

            Show
            jbosch Jim Bosch added a comment - This has passed ci_imsim in Jenkins (vanilla + ci_hsc Jenkins is still running), so I've now got some confidence that this at least does no harm, and Lee Kelvin 's tests seem to show it helps. Let's get it reviewed now while I investigate why ci_imsim wasn't broken before, since there's a good chance that investigation won't involve any code changes. PRs are https://github.com/lsst/obs_lsst/pull/319 and https://github.com/lsst/pipe_tasks/pull/538
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks Jim. Can confirm that two DC2 runs I kicked off both completed the entirety of multiVisit when using w25 and these DM-30838 ticket branches on obs_lsst and pipe_tasks. The input collection was a w25 singleFrame collection (itself built using 2.2i/defaults/DP0 as an input) so everything here is w25.

            The contract added into pipe_tasks reads well, and I have nothing more to add to it. I used the updated pipeline in my processing as detailed above, but I haven't been able to test the doMaskBrightObjects=True logic here. That said, I don't anticipate this should cause any issues at this stage. Assuming all Jenkins runs complete successfully, then this looks good to merge to me.

            Show
            lskelvin Lee Kelvin added a comment - Thanks Jim. Can confirm that two DC2 runs I kicked off both completed the entirety of multiVisit when using w25 and these DM-30838 ticket branches on obs_lsst and pipe_tasks . The input collection was a w25 singleFrame collection (itself built using 2.2i/defaults/DP0 as an input) so everything here is w25. The contract added into pipe_tasks reads well, and I have nothing more to add to it. I used the updated pipeline in my processing as detailed above, but I haven't been able to test the doMaskBrightObjects=True logic here. That said, I don't anticipate this should cause any issues at this stage. Assuming all Jenkins runs complete successfully, then this looks good to merge to me.
            Hide
            yusra Yusra AlSayyad added a comment -

            Huh, so do we know why ci_imsim passed for DM-30649? https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34453/pipeline/64 (back when only centos worked)

            Kicking off a master just to see if it catches this...

            Show
            yusra Yusra AlSayyad added a comment - Huh, so do we know why ci_imsim passed for DM-30649 ? https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34453/pipeline/64 (back when only centos worked) Kicking off a master just to see if it catches this...
            Hide
            jbosch Jim Bosch added a comment -

            My Jenkins run for this ticket failed in pipelines_check because it didn't like the contract (when running just the processCcd subset). I've created DM-30848 and moved the contract to a pipe_tasks branch for that ticket, and I'll just close the pipe_tasks PR and delete the branch for this one.

            Huh, so do we know why ci_imsim passed for DM-30649?

            We do not; I am planning to run a local ci_imsim on master to see if it makes coadds with a BRIGHT_OBJECT mask plane, and go from there.

            Show
            jbosch Jim Bosch added a comment - My Jenkins run for this ticket failed in pipelines_check because it didn't like the contract (when running just the processCcd subset). I've created DM-30848 and moved the contract to a pipe_tasks branch for that ticket, and I'll just close the pipe_tasks PR and delete the branch for this one. Huh, so do we know why ci_imsim passed for DM-30649 ? We do not; I am planning to run a local ci_imsim on master to see if it makes coadds with a BRIGHT_OBJECT mask plane, and go from there.
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            Oh nooooooooo. master ci_imsim passes, but if you look at the log:
            https://ci.lsst.codes/job/stack-os-matrix/34496/artifact/a138e20582/lsstsw/build/ci_imsim/_build.log

            [2021-06-22T20:27:18.726278Z]     Invalid mask plane name: BRIGHT_OBJECT {0}
            [2021-06-22T20:27:18.726281Z] lsst::meas::base::FatalAlgorithmError: 'Invalid mask plane name: BRIGHT_OBJECT'
            [2021-06-22T20:27:18.726284Z] 
            [2021-06-22T20:27:19.992363Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 37 failed and 15 remain out of total 338 quanta.
            [2021-06-22T20:27:20.093280Z] ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(MeasureMergedCoaddSourcesTask, label=measure) dataId={band: 'i', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}> failed; processing will continue for remaining tasks.
            [2021-06-22T20:27:20.093514Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 55, tract: 0, visit: 5884, ...}>, skipping this task.
            [2021-06-22T20:27:20.093705Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 55, tract: 0, visit: 206039, ...}>, skipping this task.
            [2021-06-22T20:27:20.093877Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 138, tract: 0, visit: 496989, ...}>, skipping this task.
            [2021-06-22T20:27:20.094052Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCoaddTask, label=forcedPhotCoadd) dataId={band: 'g', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task.
            [2021-06-22T20:27:20.094229Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 14, tract: 0, visit: 180001, ...}>, skipping this task.
            [2021-06-22T20:27:20.094377Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(TransformObjectCatalogTask, label=transformObjectTable) dataId={skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task.
            [2021-06-22T20:27:20.094712Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 44 failed and 8 remain out of total 338 quanta.
            [2021-06-22T20:27:20.203735Z] ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(MeasureMergedCoaddSourcesTask, label=measure) dataId={band: 'u', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}> failed; processing will continue for remaining tasks.
            [2021-06-22T20:27:20.203907Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 13, tract: 0, visit: 180001, ...}>, skipping this task.
            [2021-06-22T20:27:20.204080Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 82, tract: 0, visit: 277060, ...}>, skipping this task.
            [2021-06-22T20:27:20.204252Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCoaddTask, label=forcedPhotCoadd) dataId={band: 'i', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task.
            [2021-06-22T20:27:20.204489Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 48 failed and 4 remain out of total 338 quanta.
            [2021-06-22T20:27:20.306338Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 76, tract: 0, visit: 8003, ...}>, skipping this task.
            [2021-06-22T20:27:20.306490Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ConsolidateObjectTableTask, label=consolidateObjectTable) dataId={skymap: 'discrete/ci_imsim/4k', tract: 0}>, skipping this task.
            [2021-06-22T20:27:20.306687Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 50 failed and 2 remain out of total 338 quanta.
            [2021-06-22T20:27:20.407093Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCoaddTask, label=forcedPhotCoadd) dataId={band: 'z', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task.
            [2021-06-22T20:27:20.407263Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 51 failed and 1 remain out of total 338 quanta.
            [2021-06-22T20:27:24.345650Z] measure.measurement INFO: Measuring 6157 sources (6123 parents, 34 children) 
            [2021-06-22T20:27:24.523424Z] Process task-286:
            [2021-06-22T20:27:24.523453Z] Traceback (most recent call last):
            [2021-06-22T20:27:24.523458Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/miniconda/envs/lsst-scipipe-0.6.0/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
            [2021-06-22T20:27:24.523462Z]     self.run()
            [2021-06-22T20:27:24.523465Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/miniconda/envs/lsst-scipipe-0.6.0/lib/python3.8/multiprocessing/process.py", line 108, in run
            [2021-06-22T20:27:24.523468Z]     self._target(*self._args, **self._kwargs)
            [2021-06-22T20:27:24.523471Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/ctrl_mpexec/21.0.0-33-g1dfc228+d099b1da2c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 128, in _executeJob
            [2021-06-22T20:27:24.523475Z]     quantumExecutor.execute(taskDef, quantum, butler)
            [2021-06-22T20:27:24.523478Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/ctrl_mpexec/21.0.0-33-g1dfc228+d099b1da2c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 137, in execute
            [2021-06-22T20:27:24.523481Z]     self.runQuantum(task, quantum, taskDef, butler)
            [2021-06-22T20:27:24.523483Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/ctrl_mpexec/21.0.0-33-g1dfc228+d099b1da2c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 345, in runQuantum
            [2021-06-22T20:27:24.523486Z]     task.runQuantum(butlerQC, inputRefs, outputRefs)
            [2021-06-22T20:27:24.523489Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/pipe_tasks/21.0.0-108-g9e5eb3bb+d6ef70ab25/python/lsst/pipe/tasks/multiBand.py", line 996, in runQuantum
            [2021-06-22T20:27:24.523492Z]     outputs = self.run(**inputs)
            [2021-06-22T20:27:24.523495Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/pipe_tasks/21.0.0-108-g9e5eb3bb+d6ef70ab25/python/lsst/pipe/tasks/multiBand.py", line 1073, in run
            [2021-06-22T20:27:24.523498Z]     self.measurement.run(sources, exposure, exposureId=exposureId)
            [2021-06-22T20:27:24.523501Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/pipe_base/21.0.0-28-gc2696ae+031a9e42c7/python/lsst/pipe/base/timer.py", line 182, in wrapper
            [2021-06-22T20:27:24.523507Z]     res = func(self, *args, **keyArgs)
            [2021-06-22T20:27:24.523510Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/sfm.py", line 258, in run
            [2021-06-22T20:27:24.523513Z]     self.runPlugins(noiseReplacer, measCat, exposure, beginOrder, endOrder)
            [2021-06-22T20:27:24.523516Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/sfm.py", line 312, in runPlugins
            [2021-06-22T20:27:24.523519Z]     self.callMeasure(measParentRecord, exposure, beginOrder=beginOrder, endOrder=endOrder)
            [2021-06-22T20:27:24.523522Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/baseMeasurement.py", line 331, in callMeasure
            [2021-06-22T20:27:24.523525Z]     self.doMeasurement(plugin, measRecord, *args, **kwds)
            [2021-06-22T20:27:24.523528Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/baseMeasurement.py", line 361, in doMeasurement
            [2021-06-22T20:27:24.523531Z]     plugin.measure(measRecord, *args, **kwds)
            [2021-06-22T20:27:24.523533Z]   File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/wrappers.py", line 43, in measure
            [2021-06-22T20:27:24.523536Z]     self.cpp.measure(measRecord, exposure)
            [2021-06-22T20:27:24.523539Z] lsst.pex.exceptions.wrappers.FatalAlgorithmError: 
            [2021-06-22T20:27:24.523542Z]   File "src/PixelFlags.cc", line 70, in void lsst::meas::base::(anonymous namespace)::updateFlags(const PixelFlagsAlgorithm::KeyMap &, const FootprintBits<lsst::meas::base::(anonymous namespace)::MaskedImageF> &, afw::table::SourceRecord &)
            [2021-06-22T20:27:24.523545Z]     Invalid mask plane name: BRIGHT_OBJECT {0}
            [2021-06-22T20:27:24.523548Z] lsst::meas::base::FatalAlgorithmError: 'Invalid mask plane name: BRIGHT_OBJECT'
            [2021-06-22T20:27:24.523551Z] 
            

            (contacted Dan Taranu)

            Show
            yusra Yusra AlSayyad added a comment - - edited Oh nooooooooo. master ci_imsim passes, but if you look at the log: https://ci.lsst.codes/job/stack-os-matrix/34496/artifact/a138e20582/lsstsw/build/ci_imsim/_build.log [2021-06-22T20:27:18.726278Z] Invalid mask plane name: BRIGHT_OBJECT {0} [2021-06-22T20:27:18.726281Z] lsst::meas::base::FatalAlgorithmError: 'Invalid mask plane name: BRIGHT_OBJECT' [2021-06-22T20:27:18.726284Z] [2021-06-22T20:27:19.992363Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 37 failed and 15 remain out of total 338 quanta. [2021-06-22T20:27:20.093280Z] ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(MeasureMergedCoaddSourcesTask, label=measure) dataId={band: 'i', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}> failed; processing will continue for remaining tasks. [2021-06-22T20:27:20.093514Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 55, tract: 0, visit: 5884, ...}>, skipping this task. [2021-06-22T20:27:20.093705Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 55, tract: 0, visit: 206039, ...}>, skipping this task. [2021-06-22T20:27:20.093877Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 138, tract: 0, visit: 496989, ...}>, skipping this task. [2021-06-22T20:27:20.094052Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCoaddTask, label=forcedPhotCoadd) dataId={band: 'g', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task. [2021-06-22T20:27:20.094229Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 14, tract: 0, visit: 180001, ...}>, skipping this task. [2021-06-22T20:27:20.094377Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(TransformObjectCatalogTask, label=transformObjectTable) dataId={skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task. [2021-06-22T20:27:20.094712Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 44 failed and 8 remain out of total 338 quanta. [2021-06-22T20:27:20.203735Z] ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(MeasureMergedCoaddSourcesTask, label=measure) dataId={band: 'u', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}> failed; processing will continue for remaining tasks. [2021-06-22T20:27:20.203907Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 13, tract: 0, visit: 180001, ...}>, skipping this task. [2021-06-22T20:27:20.204080Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 82, tract: 0, visit: 277060, ...}>, skipping this task. [2021-06-22T20:27:20.204252Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCoaddTask, label=forcedPhotCoadd) dataId={band: 'i', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task. [2021-06-22T20:27:20.204489Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 48 failed and 4 remain out of total 338 quanta. [2021-06-22T20:27:20.306338Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCcdTask, label=forcedPhotCcd) dataId={instrument: 'LSSTCam-imSim', skymap: 'discrete/ci_imsim/4k', detector: 76, tract: 0, visit: 8003, ...}>, skipping this task. [2021-06-22T20:27:20.306490Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ConsolidateObjectTableTask, label=consolidateObjectTable) dataId={skymap: 'discrete/ci_imsim/4k', tract: 0}>, skipping this task. [2021-06-22T20:27:20.306687Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 50 failed and 2 remain out of total 338 quanta. [2021-06-22T20:27:20.407093Z] ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(ForcedPhotCoaddTask, label=forcedPhotCoadd) dataId={band: 'z', skymap: 'discrete/ci_imsim/4k', tract: 0, patch: 24}>, skipping this task. [2021-06-22T20:27:20.407263Z] ctrl.mpexec.mpGraphExecutor INFO: Executed 286 quanta successfully, 51 failed and 1 remain out of total 338 quanta. [2021-06-22T20:27:24.345650Z] measure.measurement INFO: Measuring 6157 sources (6123 parents, 34 children) [2021-06-22T20:27:24.523424Z] Process task-286: [2021-06-22T20:27:24.523453Z] Traceback (most recent call last): [2021-06-22T20:27:24.523458Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/miniconda/envs/lsst-scipipe-0.6.0/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [2021-06-22T20:27:24.523462Z] self.run() [2021-06-22T20:27:24.523465Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/miniconda/envs/lsst-scipipe-0.6.0/lib/python3.8/multiprocessing/process.py", line 108, in run [2021-06-22T20:27:24.523468Z] self._target(*self._args, **self._kwargs) [2021-06-22T20:27:24.523471Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/ctrl_mpexec/21.0.0-33-g1dfc228+d099b1da2c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 128, in _executeJob [2021-06-22T20:27:24.523475Z] quantumExecutor.execute(taskDef, quantum, butler) [2021-06-22T20:27:24.523478Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/ctrl_mpexec/21.0.0-33-g1dfc228+d099b1da2c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 137, in execute [2021-06-22T20:27:24.523481Z] self.runQuantum(task, quantum, taskDef, butler) [2021-06-22T20:27:24.523483Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/ctrl_mpexec/21.0.0-33-g1dfc228+d099b1da2c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 345, in runQuantum [2021-06-22T20:27:24.523486Z] task.runQuantum(butlerQC, inputRefs, outputRefs) [2021-06-22T20:27:24.523489Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/pipe_tasks/21.0.0-108-g9e5eb3bb+d6ef70ab25/python/lsst/pipe/tasks/multiBand.py", line 996, in runQuantum [2021-06-22T20:27:24.523492Z] outputs = self.run(**inputs) [2021-06-22T20:27:24.523495Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/pipe_tasks/21.0.0-108-g9e5eb3bb+d6ef70ab25/python/lsst/pipe/tasks/multiBand.py", line 1073, in run [2021-06-22T20:27:24.523498Z] self.measurement.run(sources, exposure, exposureId=exposureId) [2021-06-22T20:27:24.523501Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/pipe_base/21.0.0-28-gc2696ae+031a9e42c7/python/lsst/pipe/base/timer.py", line 182, in wrapper [2021-06-22T20:27:24.523507Z] res = func(self, *args, **keyArgs) [2021-06-22T20:27:24.523510Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/sfm.py", line 258, in run [2021-06-22T20:27:24.523513Z] self.runPlugins(noiseReplacer, measCat, exposure, beginOrder, endOrder) [2021-06-22T20:27:24.523516Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/sfm.py", line 312, in runPlugins [2021-06-22T20:27:24.523519Z] self.callMeasure(measParentRecord, exposure, beginOrder=beginOrder, endOrder=endOrder) [2021-06-22T20:27:24.523522Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/baseMeasurement.py", line 331, in callMeasure [2021-06-22T20:27:24.523525Z] self.doMeasurement(plugin, measRecord, *args, **kwds) [2021-06-22T20:27:24.523528Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/baseMeasurement.py", line 361, in doMeasurement [2021-06-22T20:27:24.523531Z] plugin.measure(measRecord, *args, **kwds) [2021-06-22T20:27:24.523533Z] File "/Users/square/j/ws/stack-os-matrix/a138e20582/lsstsw/stack/0.6.0/DarwinX86/meas_base/21.0.0-13-g64487bb+d13b8a0dbe/python/lsst/meas/base/wrappers.py", line 43, in measure [2021-06-22T20:27:24.523536Z] self.cpp.measure(measRecord, exposure) [2021-06-22T20:27:24.523539Z] lsst.pex.exceptions.wrappers.FatalAlgorithmError: [2021-06-22T20:27:24.523542Z] File "src/PixelFlags.cc", line 70, in void lsst::meas::base::(anonymous namespace)::updateFlags(const PixelFlagsAlgorithm::KeyMap &, const FootprintBits<lsst::meas::base::(anonymous namespace)::MaskedImageF> &, afw::table::SourceRecord &) [2021-06-22T20:27:24.523545Z] Invalid mask plane name: BRIGHT_OBJECT {0} [2021-06-22T20:27:24.523548Z] lsst::meas::base::FatalAlgorithmError: 'Invalid mask plane name: BRIGHT_OBJECT' [2021-06-22T20:27:24.523551Z] (contacted Dan Taranu )
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks for the update on Jenkins. Moving the contract aspect into its own ticket seems like a good move in which case.

            Show
            lskelvin Lee Kelvin added a comment - Thanks for the update on Jenkins. Moving the contract aspect into its own ticket seems like a good move in which case.
            Hide
            jbosch Jim Bosch added a comment -

            Thanks, Dan Taranu and Yusra AlSayyad, for figuring out the ci_imsim mystery before I managed to spend any time on it beyond "git clone". I've merged this fix, so hopefully that addresses the only thing that would be causing ci_imsim to fail if it was capable of failing.

            Show
            jbosch Jim Bosch added a comment - Thanks, Dan Taranu and Yusra AlSayyad , for figuring out the ci_imsim mystery before I managed to spend any time on it beyond "git clone". I've merged this fix, so hopefully that addresses the only thing that would be causing ci_imsim to fail if it was capable of failing.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Lee Kelvin
              Watchers:
              Jim Bosch, Lee Kelvin, Morgan Schmitz, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.