# constructDark.py fails on LSSTCam with 'Image contains no Pixels'

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Data Release Production
• Urgent?:
No

#### Description

I ran a PTC task on BOT run 12606, the first time we have the full LSSTCam.  Six CCDs failed to return results.  I am tracking down the reasons.  CCD R01_S00, detector=0 failed because it failed to create a master dark with the error "Image contains no Pixels". I have copied a command line that will reproduce the error, as well as the error itself below.  I also have a screenshot of the DS9 image of the input dark.  It contains a large bright defect, which may be the cause of the error.  I tried several other 300 second darks, including 3020100800105, but they gave the same error.

Command string:

 constructDark.py /project/shared/BOT --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12606/CALIB --rerun /project/shared/BOT/rerun/cslage/PTC_Test --batch-type=smp --cores 1 -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 --clobber-config --id expId=3020100800045 detector=0 

Error:

 dark WARN: Unable to process DataId(initialdata={'expId': 3020100800045, 'detector': 0, 'dayObs': '2020-10-08', 'raftName': 'R01', 'detectorName': 'S00', 'snap': 0}, tag=set()):   File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter; MaskT = lsst::afw::math::MaskImposter; VarianceT = lsst::afw::math::MaskImposter; WeightT = lsst::afw::math::MaskImposter]  Image contains no pixels {0} lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'InvalidParameterError on lsst-devl01:2444703 in reduce:   File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter; MaskT = lsst::afw::math::MaskImposter; VarianceT = lsst::afw::math::MaskImposter; WeightT = lsst::afw::math::MaskImposter]  Image contains no pixels {0} lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'Traceback (most recent call last):  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 112, in wrapper  return func(*args, **kwargs)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 239, in wrapper  return func(*args, **kwargs)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 715, in reduce  *args, **kwargs)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 572, in _reduceQueue  resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 572, in   resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 625, in process  exposure = self.processSingle(sensorRef, **kwargs)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 983, in processSingle  exposure = CalibTask.processSingle(self, sensorRef)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 644, in processSingle  return self.isr.runDataRef(dataRef).exposure  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef  result = self.run(ccdExposure, camera=camera, **isrData.getDict())  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 1366, in run  overscanResults = self.overscanCorrection(ccdExposure, amp)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 2008, in overscanCorrection  overscanResults = self.overscan.run(ampImage.getImage(), overscanImage)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 140, in run  overscanResult = self.measureVectorOverscan(overscanImage)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 459, in measureVectorOverscan  overscanVector = self.collapseArrayMedian(masked)  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 345, in collapseArrayMedian  rowMedian = afwMath.makeStatistics(newRow, fitType, self.statControl).getValue() lsst.pex.exceptions.wrappers.InvalidParameterError:   File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter; MaskT = lsst::afw::math::MaskImposter; VarianceT = lsst::afw::math::MaskImposter; WeightT = lsst::afw::math::MaskImposter]  Image contains no pixels {0} lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels' 

#### Attachments

1. 3020100800045-R01-S00.png
357 kB

#### Activity

Hide
Christopher Waters added a comment -

This appears to be a failure mode of the overscan code.  Outliers are found by calculating statistics over the entire overscan region, with discrepant points masked then.  For this camera, the overscan is then collapsed along the serial direction by taking the MEDIAN_PER_ROW.  If all the points in that row have been masked, then the statistic raises. For the image in the example, this was occuring on amp C14.

An updated ip_isr branch catches this failure, and sets the overscan to NaN (which will result in masking downstream, but no other value makes sense here).

Show
Christopher Waters added a comment - This appears to be a failure mode of the overscan code.  Outliers are found by calculating statistics over the entire overscan region, with discrepant points masked then.  For this camera, the overscan is then collapsed along the serial direction by taking the MEDIAN_PER_ROW.  If all the points in that row have been masked, then the statistic raises. For the image in the example, this was occuring on amp C14. An updated ip_isr branch catches this failure, and sets the overscan to NaN (which will result in masking downstream, but no other value makes sense here).
Hide
Craig Lage added a comment -

Yes, it runs with isr.overscan.fitType=MEDIAN instead of MEDIAN_PER_ROW.  I'll run this way for now.

Show
Craig Lage added a comment - Yes, it runs with isr.overscan.fitType=MEDIAN instead of MEDIAN_PER_ROW.  I'll run this way for now.
Hide
Craig Lage added a comment -

I accidentally blew away that repo, so that CALIB directory is gone.  I thought clobber-ouput only deleted what was being changed, but it deletes everything!.  I'm re-building the repo now, but it will take a while.

Show
Craig Lage added a comment - I accidentally blew away that repo, so that CALIB directory is gone.  I thought clobber-ouput only deleted what was being changed, but it deletes everything!.  I'm re-building the repo now, but it will take a while.
Hide
Christopher Waters added a comment -

I'm testing the ip_isr branch with the fix now.  It should be into and possibly through review tomorrow.

Show
Christopher Waters added a comment - I'm testing the ip_isr branch with the fix now.  It should be into and possibly through review tomorrow.
Hide
Christopher Waters added a comment -
Show
Christopher Waters added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32851/pipeline
Hide
Andrés Alejandro Plazas Malagón added a comment -

I think this looks good; I just asked for some minor clarifications, especially in the part that seems to be doing the fix for this ticket (len(newRow)). As a more general and broad comment, it seems that the debug part is always interactive (i.e., it waits for the user to provide input before continuing). Is this correct? This is the impression that I get from other debug options in other tasks (e.g, in crosstalk). If we want to run a script over many detectors and with multiple tasks, for example, and we also want to look at the debug plots, would that be possible without the interaction with the user?

Show
Andrés Alejandro Plazas Malagón added a comment - I think this looks good; I just asked for some minor clarifications, especially in the part that seems to be doing the fix for this ticket ( len(newRow) ). As a more general and broad comment, it seems that the debug part is always interactive (i.e., it waits for the user to provide input before continuing). Is this correct? This is the impression that I get from other debug options in other tasks (e.g, in crosstalk). If we want to run a script over many detectors and with multiple tasks, for example, and we also want to look at the debug plots, would that be possible without the interaction with the user?

#### People

Assignee:
Christopher Waters
Reporter:
Craig Lage
Reviewers:
Andrés Alejandro Plazas Malagón
Watchers:
Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage