[DM-33633] Verification pipetask at the summit (via Nublado terminal) fails with "-j 8" option but works without it - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: ctrl_mpexec
Labels:
- gen3-middleware

Story Points:
1
Epic Link:
S22 Gen3 Middleware
Sprint:
DB_S22_12
Team:
Data Access and Database
Urgent?:
No

Description

Running the verification bias pipetask on a terminal in the summit nublado (with d_2022_02_09 and pipe_base locally set up) gives error messages without log messages when setting the option -j 8. However, the task successfully runs when getting rid of that option.

Outputs for runs that failed and that successfully finished are attached as text files.

Hide

Permalink

Andy Salnikov added a comment - 10/Feb/22 8:33 PM

Does it also fail with -j 2 instead of -j 8?

Show

Andy Salnikov added a comment - 10/Feb/22 8:33 PM Does it also fail with -j 2 instead of -j 8 ?

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 10/Feb/22 8:52 PM

It worked with -j 2:

[plazas@nb-plazas pipe_base]$ pipetask --long-log run -p '${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml' -b /repo/LATISS -o u/plazas/LATISS-test-2022FEB10.7 -i LATISS/raw/all -j 2 -i u/ocps/23c2f95642844f5594128745688165c1,LATISS/calib --register-dataset-types --output-run u/plazas/LATISS-test-2022FEB10.7/run -d 'instrument='\''LATISS'\'' AND detector IN (0) AND exposure IN (2022020800124, 2022020800125, 2022020800126, 2022020800127, 2022020800128)'

INFO 2022-02-10T20:45:35.871+00:00 numexpr.utils ()(utils.py:147) - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.

INFO 2022-02-10T20:45:35.872+00:00 numexpr.utils ()(utils.py:159) - NumExpr defaulting to 8 threads.

WARNING 2022-02-10T20:45:36.390+00:00 lsst.pipe.base.graphBuilder ()(graphBuilder.py:157) - Dataset type mismatch (DatasetType('verifyBiasApply_metadata', {band, instrument, detector, physical_filter, exposure}, PropertySet) != DatasetType('verifyBiasApply_metadata', {band, instrument, detector, physical_filter, exposure}, TaskMetadata)) but continuing since they are compatible

INFO 2022-02-10T20:45:36.709+00:00 lsst.ctrl.mpexec.cmdLineFwk ()(cmdLineFwk.py:614) - QuantumGraph contains 16 quanta for 4 tasks, graph ID: '1644525936.707119-6774'

INFO 2022-02-10T20:45:38.842+00:00 conda.common.io ()(io.py:246) - overtaking stderr and stdout

INFO 2022-02-10T20:45:43.582+00:00 conda.core.prefix_data ()(prefix_data.py:288) - Python record ignored for anchor path 'lib/python3.8/site-packages/ts-idl.egg-link'

  due to [Errno 2] No such file or directory: '/home/saluser/miniconda3/conda-bld/ts-idl_1642561654639/work/python'

INFO 2022-02-10T20:45:43.701+00:00 conda.common.io ()(io.py:261) - stderr and stdout yielding back

WARNING 2022-02-10T20:45:44.277+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000126-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047)))

WARNING 2022-02-10T20:45:44.278+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000128-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047)))

INFO 2022-02-10T20:45:48.088+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1453) - Converting exposure to floating point values.

INFO 2022-02-10T20:45:48.089+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1453) - Converting exposure to floating point values.

INFO 2022-02-10T20:45:51.470+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1520) - Assembling CCD from amplifiers.

INFO 2022-02-10T20:45:51.480+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1520) - Assembling CCD from amplifiers.

WARNING 2022-02-10T20:45:51.676+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1524) - No WCS found in input exposure.

INFO 2022-02-10T20:45:51.677+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1532) - Applying bias correction.

WARNING 2022-02-10T20:45:51.693+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1524) - No WCS found in input exposure.

INFO 2022-02-10T20:45:51.693+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1532) - Applying bias correction.

INFO 2022-02-10T20:45:52.798+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1578) - Masking defects.

INFO 2022-02-10T20:45:52.799+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels.

INFO 2022-02-10T20:45:52.862+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1578) - Masking defects.

INFO 2022-02-10T20:45:52.864+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels.

INFO 2022-02-10T20:45:52.869+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1591) - Widening saturation trails.

INFO 2022-02-10T20:45:52.882+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1591) - Widening saturation trails.

INFO 2022-02-10T20:45:53.868+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1735) - Set 8699 BAD pixels to -0.020996.

INFO 2022-02-10T20:45:53.868+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1738) - Interpolating masked pixels.

INFO 2022-02-10T20:45:53.904+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1735) - Set 8696 BAD pixels to -0.033203.

INFO 2022-02-10T20:45:53.904+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1738) - Interpolating masked pixels.

WARNING 2022-02-10T20:45:55.101+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty.

WARNING 2022-02-10T20:45:55.101+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point.

WARNING 2022-02-10T20:45:55.206+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty.

WARNING 2022-02-10T20:45:55.207+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point.

INFO 2022-02-10T20:45:58.096+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...} took 14.222 seconds

INFO 2022-02-10T20:45:58.205+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 1 quanta successfully, 0 failed and 15 remain out of total 16 quanta.

WARNING 2022-02-10T20:45:58.610+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000127-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047)))

INFO 2022-02-10T20:46:02.375+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1453) - Converting exposure to floating point values.

INFO 2022-02-10T20:46:03.497+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...} took 19.616 seconds

INFO 2022-02-10T20:46:03.660+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 2 quanta successfully, 0 failed and 14 remain out of total 16 quanta.

WARNING 2022-02-10T20:46:04.189+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000124-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047)))

INFO 2022-02-10T20:46:05.385+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1520) - Assembling CCD from amplifiers.

WARNING 2022-02-10T20:46:05.586+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1524) - No WCS found in input exposure.

INFO 2022-02-10T20:46:05.586+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1532) - Applying bias correction.

INFO 2022-02-10T20:46:06.701+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1578) - Masking defects.

INFO 2022-02-10T20:46:06.703+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels.

INFO 2022-02-10T20:46:06.772+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1591) - Widening saturation trails.

INFO 2022-02-10T20:46:07.669+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1735) - Set 6448 BAD pixels to 0.022135.

INFO 2022-02-10T20:46:07.670+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1738) - Interpolating masked pixels.

INFO 2022-02-10T20:46:07.893+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1453) - Converting exposure to floating point values.

WARNING 2022-02-10T20:46:08.998+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty.

WARNING 2022-02-10T20:46:08.999+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point.

INFO 2022-02-10T20:46:11.290+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1520) - Assembling CCD from amplifiers.

WARNING 2022-02-10T20:46:11.484+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1524) - No WCS found in input exposure.

INFO 2022-02-10T20:46:11.484+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1532) - Applying bias correction.

INFO 2022-02-10T20:46:12.612+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1578) - Masking defects.

INFO 2022-02-10T20:46:12.661+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels.

INFO 2022-02-10T20:46:12.680+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1591) - Widening saturation trails.

INFO 2022-02-10T20:46:13.595+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1735) - Set 8262 BAD pixels to 0.016796.

INFO 2022-02-10T20:46:13.596+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1738) - Interpolating masked pixels.

INFO 2022-02-10T20:46:14.503+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...} took 16.295 seconds

INFO 2022-02-10T20:46:14.684+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 3 quanta successfully, 0 failed and 13 remain out of total 16 quanta.

WARNING 2022-02-10T20:46:14.771+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty.

WARNING 2022-02-10T20:46:14.771+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point.

WARNING 2022-02-10T20:46:15.180+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000125-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047)))

INFO 2022-02-10T20:46:18.870+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1453) - Converting exposure to floating point values.

INFO 2022-02-10T20:46:20.104+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...} took 16.441 seconds

INFO 2022-02-10T20:46:20.200+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 4 quanta successfully, 0 failed and 12 remain out of total 16 quanta.

INFO 2022-02-10T20:46:21.970+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1520) - Assembling CCD from amplifiers.

WARNING 2022-02-10T20:46:22.166+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1524) - No WCS found in input exposure.

INFO 2022-02-10T20:46:22.167+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1532) - Applying bias correction.

INFO 2022-02-10T20:46:23.293+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1578) - Masking defects.

INFO 2022-02-10T20:46:23.294+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels.

INFO 2022-02-10T20:46:23.363+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1591) - Widening saturation trails.

INFO 2022-02-10T20:46:24.265+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1735) - Set 6452 BAD pixels to 0.027669.

INFO 2022-02-10T20:46:24.266+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1738) - Interpolating masked pixels.

INFO 2022-02-10T20:46:24.692+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(repair.py:292) - Identified 0 cosmic rays.

INFO 2022-02-10T20:46:24.989+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...} took 4.785 seconds

INFO 2022-02-10T20:46:25.114+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 5 quanta successfully, 0 failed and 11 remain out of total 16 quanta.

WARNING 2022-02-10T20:46:25.403+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty.

WARNING 2022-02-10T20:46:25.404+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point.

INFO 2022-02-10T20:46:29.970+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(repair.py:292) - Identified 4 cosmic rays.

INFO 2022-02-10T20:46:30.651+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...} took 15.964 seconds

INFO 2022-02-10T20:46:30.774+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 6 quanta successfully, 0 failed and 10 remain out of total 16 quanta.

INFO 2022-02-10T20:46:30.870+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...} took 5.753 seconds

INFO 2022-02-10T20:46:30.983+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 7 quanta successfully, 0 failed and 9 remain out of total 16 quanta.

INFO 2022-02-10T20:46:35.192+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(repair.py:292) - Identified 4 cosmic rays.

INFO 2022-02-10T20:46:35.369+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(repair.py:292) - Identified 8 cosmic rays.

INFO 2022-02-10T20:46:35.496+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...} took 4.719 seconds

INFO 2022-02-10T20:46:35.599+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 8 quanta successfully, 0 failed and 8 remain out of total 16 quanta.

INFO 2022-02-10T20:46:35.660+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...} took 4.674 seconds

INFO 2022-02-10T20:46:35.809+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 9 quanta successfully, 0 failed and 7 remain out of total 16 quanta.

INFO 2022-02-10T20:46:36.112+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800126, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800126, ...} took 0.300 seconds

INFO 2022-02-10T20:46:36.218+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 10 quanta successfully, 0 failed and 6 remain out of total 16 quanta.

INFO 2022-02-10T20:46:36.580+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800127, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800127, ...} took 0.359 seconds

INFO 2022-02-10T20:46:36.624+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 11 quanta successfully, 0 failed and 5 remain out of total 16 quanta.

INFO 2022-02-10T20:46:36.980+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800128, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800128, ...} took 0.317 seconds

INFO 2022-02-10T20:46:37.069+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 12 quanta successfully, 0 failed and 4 remain out of total 16 quanta.

INFO 2022-02-10T20:46:37.374+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800124, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800124, ...} took 0.303 seconds

INFO 2022-02-10T20:46:37.470+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 13 quanta successfully, 0 failed and 3 remain out of total 16 quanta.

INFO 2022-02-10T20:46:38.698+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(repair.py:292) - Identified 3 cosmic rays.

INFO 2022-02-10T20:46:38.877+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...} took 3.275 seconds

INFO 2022-02-10T20:46:38.981+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 14 quanta successfully, 0 failed and 2 remain out of total 16 quanta.

INFO 2022-02-10T20:46:39.173+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800125, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800125, ...} took 0.190 seconds

INFO 2022-02-10T20:46:39.289+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 15 quanta successfully, 0 failed and 1 remain out of total 16 quanta.

INFO 2022-02-10T20:46:39.574+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyRunStats:{instrument: 'LATISS'})(singleQuantumExecutor.py:209) - Execution of task 'verifyRunStats' on quantum {instrument: 'LATISS'} took 0.282 seconds

INFO 2022-02-10T20:46:39.691+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 16 quanta successfully, 0 failed and 0 remain out of total 16 quanta.

Show

Andrés Alejandro Plazas Malagón added a comment - 10/Feb/22 8:52 PM It worked with -j 2 : [plazas@nb-plazas pipe_base]$ pipetask --long-log run -p '${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml' -b /repo/LATISS -o u/plazas/LATISS-test-2022FEB10.7 -i LATISS/raw/all -j 2 -i u/ocps/23c2f95642844f5594128745688165c1,LATISS/calib --register-dataset-types --output-run u/plazas/LATISS-test-2022FEB10.7/run -d 'instrument='\''LATISS'\'' AND detector IN (0) AND exposure IN (2022020800124, 2022020800125, 2022020800126, 2022020800127, 2022020800128)' INFO 2022-02-10T20:45:35.871+00:00 numexpr.utils ()(utils.py:147) - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. INFO 2022-02-10T20:45:35.872+00:00 numexpr.utils ()(utils.py:159) - NumExpr defaulting to 8 threads. WARNING 2022-02-10T20:45:36.390+00:00 lsst.pipe.base.graphBuilder ()(graphBuilder.py:157) - Dataset type mismatch (DatasetType('verifyBiasApply_metadata', {band, instrument, detector, physical_filter, exposure}, PropertySet) != DatasetType('verifyBiasApply_metadata', {band, instrument, detector, physical_filter, exposure}, TaskMetadata)) but continuing since they are compatible INFO 2022-02-10T20:45:36.709+00:00 lsst.ctrl.mpexec.cmdLineFwk ()(cmdLineFwk.py:614) - QuantumGraph contains 16 quanta for 4 tasks, graph ID: '1644525936.707119-6774' INFO 2022-02-10T20:45:38.842+00:00 conda.common.io ()(io.py:246) - overtaking stderr and stdout INFO 2022-02-10T20:45:43.582+00:00 conda.core.prefix_data ()(prefix_data.py:288) - Python record ignored for anchor path 'lib/python3.8/site-packages/ts-idl.egg-link' due to [Errno 2] No such file or directory: '/home/saluser/miniconda3/conda-bld/ts-idl_1642561654639/work/python' INFO 2022-02-10T20:45:43.701+00:00 conda.common.io ()(io.py:261) - stderr and stdout yielding back WARNING 2022-02-10T20:45:44.277+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000126-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047))) WARNING 2022-02-10T20:45:44.278+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000128-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047))) INFO 2022-02-10T20:45:48.088+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1453) - Converting exposure to floating point values. INFO 2022-02-10T20:45:48.089+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1453) - Converting exposure to floating point values. INFO 2022-02-10T20:45:51.470+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1520) - Assembling CCD from amplifiers. INFO 2022-02-10T20:45:51.480+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1520) - Assembling CCD from amplifiers. WARNING 2022-02-10T20:45:51.676+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1524) - No WCS found in input exposure. INFO 2022-02-10T20:45:51.677+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1532) - Applying bias correction. WARNING 2022-02-10T20:45:51.693+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1524) - No WCS found in input exposure. INFO 2022-02-10T20:45:51.693+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1532) - Applying bias correction. INFO 2022-02-10T20:45:52.798+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1578) - Masking defects. INFO 2022-02-10T20:45:52.799+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels. INFO 2022-02-10T20:45:52.862+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1578) - Masking defects. INFO 2022-02-10T20:45:52.864+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels. INFO 2022-02-10T20:45:52.869+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1591) - Widening saturation trails. INFO 2022-02-10T20:45:52.882+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1591) - Widening saturation trails. INFO 2022-02-10T20:45:53.868+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1735) - Set 8699 BAD pixels to -0.020996. INFO 2022-02-10T20:45:53.868+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:1738) - Interpolating masked pixels. INFO 2022-02-10T20:45:53.904+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1735) - Set 8696 BAD pixels to -0.033203. INFO 2022-02-10T20:45:53.904+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:1738) - Interpolating masked pixels. WARNING 2022-02-10T20:45:55.101+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty. WARNING 2022-02-10T20:45:55.101+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point. WARNING 2022-02-10T20:45:55.206+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty. WARNING 2022-02-10T20:45:55.207+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point. INFO 2022-02-10T20:45:58.096+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...} took 14.222 seconds INFO 2022-02-10T20:45:58.205+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 1 quanta successfully, 0 failed and 15 remain out of total 16 quanta. WARNING 2022-02-10T20:45:58.610+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000127-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047))) INFO 2022-02-10T20:46:02.375+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1453) - Converting exposure to floating point values. INFO 2022-02-10T20:46:03.497+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...} took 19.616 seconds INFO 2022-02-10T20:46:03.660+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 2 quanta successfully, 0 failed and 14 remain out of total 16 quanta. WARNING 2022-02-10T20:46:04.189+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000124-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047))) INFO 2022-02-10T20:46:05.385+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1520) - Assembling CCD from amplifiers. WARNING 2022-02-10T20:46:05.586+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1524) - No WCS found in input exposure. INFO 2022-02-10T20:46:05.586+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1532) - Applying bias correction. INFO 2022-02-10T20:46:06.701+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1578) - Masking defects. INFO 2022-02-10T20:46:06.703+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels. INFO 2022-02-10T20:46:06.772+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1591) - Widening saturation trails. INFO 2022-02-10T20:46:07.669+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1735) - Set 6448 BAD pixels to 0.022135. INFO 2022-02-10T20:46:07.670+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:1738) - Interpolating masked pixels. INFO 2022-02-10T20:46:07.893+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1453) - Converting exposure to floating point values. WARNING 2022-02-10T20:46:08.998+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty. WARNING 2022-02-10T20:46:08.999+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point. INFO 2022-02-10T20:46:11.290+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1520) - Assembling CCD from amplifiers. WARNING 2022-02-10T20:46:11.484+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1524) - No WCS found in input exposure. INFO 2022-02-10T20:46:11.484+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1532) - Applying bias correction. INFO 2022-02-10T20:46:12.612+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1578) - Masking defects. INFO 2022-02-10T20:46:12.661+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels. INFO 2022-02-10T20:46:12.680+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1591) - Widening saturation trails. INFO 2022-02-10T20:46:13.595+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1735) - Set 8262 BAD pixels to 0.016796. INFO 2022-02-10T20:46:13.596+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:1738) - Interpolating masked pixels. INFO 2022-02-10T20:46:14.503+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...} took 16.295 seconds INFO 2022-02-10T20:46:14.684+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 3 quanta successfully, 0 failed and 13 remain out of total 16 quanta. WARNING 2022-02-10T20:46:14.771+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty. WARNING 2022-02-10T20:46:14.771+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point. WARNING 2022-02-10T20:46:15.180+00:00 lsst.obs.lsst.assembly (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(assembly.py:230) - /data/lsstdata/base/auxtel/oods/gen3butler/raw/2022-02-08/AT_O_20220208_000125-R00S00.fits: outAmp.getRawBBox() != data.getBBox(); patching. ((minimum=(0, 0), maximum=(543, 2047)) v. (minimum=(0, 0), maximum=(575, 2047))) INFO 2022-02-10T20:46:18.870+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1453) - Converting exposure to floating point values. INFO 2022-02-10T20:46:20.104+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...} took 16.441 seconds INFO 2022-02-10T20:46:20.200+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 4 quanta successfully, 0 failed and 12 remain out of total 16 quanta. INFO 2022-02-10T20:46:21.970+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1520) - Assembling CCD from amplifiers. WARNING 2022-02-10T20:46:22.166+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1524) - No WCS found in input exposure. INFO 2022-02-10T20:46:22.167+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1532) - Applying bias correction. INFO 2022-02-10T20:46:23.293+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1578) - Masking defects. INFO 2022-02-10T20:46:23.294+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1587) - Masking non-finite (NAN, inf) value pixels. INFO 2022-02-10T20:46:23.363+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1591) - Widening saturation trails. INFO 2022-02-10T20:46:24.265+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1735) - Set 6452 BAD pixels to 0.027669. INFO 2022-02-10T20:46:24.266+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:1738) - Interpolating masked pixels. INFO 2022-02-10T20:46:24.692+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(repair.py:292) - Identified 0 cosmic rays. INFO 2022-02-10T20:46:24.989+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800126, ...} took 4.785 seconds INFO 2022-02-10T20:46:25.114+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 5 quanta successfully, 0 failed and 11 remain out of total 16 quanta. WARNING 2022-02-10T20:46:25.403+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:2647) - No rough magnitude zero point defined for filter SDSSr~empty. WARNING 2022-02-10T20:46:25.404+00:00 lsst.verifyBiasApply (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(isrTask.py:2652) - Non-positive exposure time; skipping rough zero point. INFO 2022-02-10T20:46:29.970+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(repair.py:292) - Identified 4 cosmic rays. INFO 2022-02-10T20:46:30.651+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyBiasApply:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyBiasApply' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...} took 15.964 seconds INFO 2022-02-10T20:46:30.774+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 6 quanta successfully, 0 failed and 10 remain out of total 16 quanta. INFO 2022-02-10T20:46:30.870+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800127, ...} took 5.753 seconds INFO 2022-02-10T20:46:30.983+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 7 quanta successfully, 0 failed and 9 remain out of total 16 quanta. INFO 2022-02-10T20:46:35.192+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(repair.py:292) - Identified 4 cosmic rays. INFO 2022-02-10T20:46:35.369+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(repair.py:292) - Identified 8 cosmic rays. INFO 2022-02-10T20:46:35.496+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800128, ...} took 4.719 seconds INFO 2022-02-10T20:46:35.599+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 8 quanta successfully, 0 failed and 8 remain out of total 16 quanta. INFO 2022-02-10T20:46:35.660+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800124, ...} took 4.674 seconds INFO 2022-02-10T20:46:35.809+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 9 quanta successfully, 0 failed and 7 remain out of total 16 quanta. INFO 2022-02-10T20:46:36.112+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800126, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800126, ...} took 0.300 seconds INFO 2022-02-10T20:46:36.218+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 10 quanta successfully, 0 failed and 6 remain out of total 16 quanta. INFO 2022-02-10T20:46:36.580+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800127, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800127, ...} took 0.359 seconds INFO 2022-02-10T20:46:36.624+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 11 quanta successfully, 0 failed and 5 remain out of total 16 quanta. INFO 2022-02-10T20:46:36.980+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800128, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800128, ...} took 0.317 seconds INFO 2022-02-10T20:46:37.069+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 12 quanta successfully, 0 failed and 4 remain out of total 16 quanta. INFO 2022-02-10T20:46:37.374+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800124, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800124, ...} took 0.303 seconds INFO 2022-02-10T20:46:37.470+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 13 quanta successfully, 0 failed and 3 remain out of total 16 quanta. INFO 2022-02-10T20:46:38.698+00:00 lsst.verifyMeasureStats.repair (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(repair.py:292) - Identified 3 cosmic rays. INFO 2022-02-10T20:46:38.877+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyMeasureStats:{instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyMeasureStats' on quantum {instrument: 'LATISS', detector: 0, exposure: 2022020800125, ...} took 3.275 seconds INFO 2022-02-10T20:46:38.981+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 14 quanta successfully, 0 failed and 2 remain out of total 16 quanta. INFO 2022-02-10T20:46:39.173+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyExposureStats:{instrument: 'LATISS', exposure: 2022020800125, ...})(singleQuantumExecutor.py:209) - Execution of task 'verifyExposureStats' on quantum {instrument: 'LATISS', exposure: 2022020800125, ...} took 0.190 seconds INFO 2022-02-10T20:46:39.289+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 15 quanta successfully, 0 failed and 1 remain out of total 16 quanta. INFO 2022-02-10T20:46:39.574+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyRunStats:{instrument: 'LATISS'})(singleQuantumExecutor.py:209) - Execution of task 'verifyRunStats' on quantum {instrument: 'LATISS'} took 0.282 seconds INFO 2022-02-10T20:46:39.691+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:498) - Executed 16 quanta successfully, 0 failed and 0 remain out of total 16 quanta.

Hide

Permalink

Andy Salnikov added a comment - 10/Feb/22 9:13 PM

Hmm, if it runs with -j2 that probably means that you run out of resources (memory) with 8 jobs and jobs either die or get killed by OOM Killer. I need to think what kind of diagnostics we can provide in this case, currently we expect that process says something before it dies, but if process is killed then it does not have a chance to say anything, and we do not handle this reasonably in parent process. I need to improve that logic and report something in addition to just saying that process has failed. I'll assign the ticket to myself and will work on that.

Show

Andy Salnikov added a comment - 10/Feb/22 9:13 PM Hmm, if it runs with -j2 that probably means that you run out of resources (memory) with 8 jobs and jobs either die or get killed by OOM Killer. I need to think what kind of diagnostics we can provide in this case, currently we expect that process says something before it dies, but if process is killed then it does not have a chance to say anything, and we do not handle this reasonably in parent process. I need to improve that logic and report something in addition to just saying that process has failed. I'll assign the ticket to myself and will work on that.

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 11/Feb/22 2:02 AM

I see, it makes sense. I was just testing this command from a terminal, but the pipetask is intended to be run by the OCPS from a calibrations script I wrote. I don't think I've seen this type of error when running the script, which has -j 8 as a default option when running pipetasks (not only the verification pipetasks). Could it be that the OCPS automatically handles memory issues?

In any case, thank you!

Show

Andrés Alejandro Plazas Malagón added a comment - 11/Feb/22 2:02 AM I see, it makes sense. I was just testing this command from a terminal, but the pipetask is intended to be run by the OCPS from a calibrations script I wrote. I don't think I've seen this type of error when running the script, which has -j 8 as a default option when running pipetasks (not only the verification pipetasks). Could it be that the OCPS automatically handles memory issues? In any case, thank you!

Hide

Permalink

Tim Jenness added a comment - 11/Feb/22 5:13 AM

Andy Salnikov yes, it's just dawned on me that this was a nublado terminal instance and those have limited memory and limited CPU cores so using -j 8 is going to be a disaster and the processes will definitely be killed with prejudice. OCPS does not have the same limitation as nublado.

Show

Tim Jenness added a comment - 11/Feb/22 5:13 AM Andy Salnikov yes, it's just dawned on me that this was a nublado terminal instance and those have limited memory and limited CPU cores so using -j 8 is going to be a disaster and the processes will definitely be killed with prejudice. OCPS does not have the same limitation as nublado.

Hide

Permalink

Andy Salnikov added a comment - 11/Feb/22 7:39 PM

Tim Jenness, could you check PR when you have a minute. This of course is not a fix for crashing jobs, just adds some info to a log message to help identify what happens, it will now say something like "Job blah failed, killed by signal 9..."

Show

Andy Salnikov added a comment - 11/Feb/22 7:39 PM Tim Jenness , could you check PR when you have a minute. This of course is not a fix for crashing jobs, just adds some info to a log message to help identify what happens, it will now say something like "Job blah failed, killed by signal 9..."

Hide

Permalink

Tim Jenness added a comment - 11/Feb/22 7:41 PM

Looks great to me. This will help a lot with diagnosis.

Show

Tim Jenness added a comment - 11/Feb/22 7:41 PM Looks great to me. This will help a lot with diagnosis.

People

Assignee:: Andy Salnikov

Reporter:: Andrés Alejandro Plazas Malagón

Reviewers:: Tim Jenness

Watchers:: Andrés Alejandro Plazas Malagón, Andy Salnikov, Christopher Waters, Tim Jenness

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 10/Feb/22 7:56 PM

Updated:: 11/Feb/22 10:04 PM

Resolved:: 11/Feb/22 9:48 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Attachments

Activity

People

Dates

Jenkins