[DM-27157] constructDark.py fails on LSSTCam with 'Image contains no Pixels' - Jira

Details

Type: Bug
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: pipe_drivers
Labels:
None

Story Points:
1
Epic Link:
f20b-drp-isr
Team:
Data Release Production
Urgent?:
No

Description

I ran a PTC task on BOT run 12606, the first time we have the full LSSTCam. Six CCDs failed to return results. I am tracking down the reasons. CCD R01_S00, detector=0 failed because it failed to create a master dark with the error "Image contains no Pixels". I have copied a command line that will reproduce the error, as well as the error itself below. I also have a screenshot of the DS9 image of the input dark. It contains a large bright defect, which may be the cause of the error. I tried several other 300 second darks, including 3020100800105, but they gave the same error.

Command string:

constructDark.py /project/shared/BOT --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12606/CALIB --rerun /project/shared/BOT/rerun/cslage/PTC_Test  --batch-type=smp  --cores 1  -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1  --clobber-config --id expId=3020100800045 detector=0

Error:

dark WARN: Unable to process DataId(initialdata={'expId': 3020100800045, 'detector': 0, 'dayObs': '2020-10-08', 'raftName': 'R01', 'detectorName': 'S00', 'snap': 0}, tag=set()):

  File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter<short unsigned int>; MaskT = lsst::afw::math::MaskImposter<int>; VarianceT = lsst::afw::math::MaskImposter<float>; WeightT = lsst::afw::math::MaskImposter<float>]

    Image contains no pixels {0}

lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'InvalidParameterError on lsst-devl01:2444703 in reduce:

  File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter<short unsigned int>; MaskT = lsst::afw::math::MaskImposter<int>; VarianceT = lsst::afw::math::MaskImposter<float>; WeightT = lsst::afw::math::MaskImposter<float>]

    Image contains no pixels {0}

lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'Traceback (most recent call last):

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 112, in wrapper

    return func(*args, **kwargs)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 239, in wrapper

    return func(*args, **kwargs)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 715, in reduce

    *args, **kwargs)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 572, in _reduceQueue

    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 572, in <listcomp>

    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 625, in process

    exposure = self.processSingle(sensorRef, **kwargs)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 983, in processSingle

    exposure = CalibTask.processSingle(self, sensorRef)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 644, in processSingle

    return self.isr.runDataRef(dataRef).exposure

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper

    res = func(self, *args, **keyArgs)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef

    result = self.run(ccdExposure, camera=camera, **isrData.getDict())

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper

    res = func(self, *args, **keyArgs)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 1366, in run

    overscanResults = self.overscanCorrection(ccdExposure, amp)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 2008, in overscanCorrection

    overscanResults = self.overscan.run(ampImage.getImage(), overscanImage)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 140, in run

    overscanResult = self.measureVectorOverscan(overscanImage)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 459, in measureVectorOverscan

    overscanVector = self.collapseArrayMedian(masked)

  File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 345, in collapseArrayMedian

    rowMedian = afwMath.makeStatistics(newRow, fitType, self.statControl).getValue()

lsst.pex.exceptions.wrappers.InvalidParameterError:

  File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter<short unsigned int>; MaskT = lsst::afw::math::MaskImposter<int>; VarianceT = lsst::afw::math::MaskImposter<float>; WeightT = lsst::afw::math::MaskImposter<float>]

    Image contains no pixels {0}

lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

3020100800045-R01-S00.png
357 kB
13/Oct/20 1:35 PM

Issue Links

blocks

DM-27161 PTC BOT issues.

Done

contains

DM-26281 Improve overscan debug information

Invalid

mentioned in: Page Loading...

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Christopher Waters added a comment - 13/Oct/20 9:43 PM

This appears to be a failure mode of the overscan code. Outliers are found by calculating statistics over the entire overscan region, with discrepant points masked then. For this camera, the overscan is then collapsed along the serial direction by taking the MEDIAN_PER_ROW. If all the points in that row have been masked, then the statistic raises. For the image in the example, this was occuring on amp C14.

An updated ip_isr branch catches this failure, and sets the overscan to NaN (which will result in masking downstream, but no other value makes sense here).

Show

Christopher Waters added a comment - 13/Oct/20 9:43 PM This appears to be a failure mode of the overscan code. Outliers are found by calculating statistics over the entire overscan region, with discrepant points masked then. For this camera, the overscan is then collapsed along the serial direction by taking the MEDIAN_PER_ROW. If all the points in that row have been masked, then the statistic raises. For the image in the example, this was occuring on amp C14. An updated ip_isr branch catches this failure, and sets the overscan to NaN (which will result in masking downstream, but no other value makes sense here).

Hide

Permalink

Craig Lage added a comment - 13/Oct/20 11:30 PM

Yes, it runs with isr.overscan.fitType=MEDIAN instead of MEDIAN_PER_ROW. I'll run this way for now.

Show

Craig Lage added a comment - 13/Oct/20 11:30 PM Yes, it runs with isr.overscan.fitType=MEDIAN instead of MEDIAN_PER_ROW. I'll run this way for now.

Hide

Permalink

Craig Lage added a comment - 13/Oct/20 11:33 PM

I accidentally blew away that repo, so that CALIB directory is gone. I thought clobber-ouput only deleted what was being changed, but it deletes everything!. I'm re-building the repo now, but it will take a while.

Show

Craig Lage added a comment - 13/Oct/20 11:33 PM I accidentally blew away that repo, so that CALIB directory is gone. I thought clobber-ouput only deleted what was being changed, but it deletes everything!. I'm re-building the repo now, but it will take a while.

Hide

Permalink

Christopher Waters added a comment - 13/Oct/20 11:34 PM

I'm testing the `ip_isr` branch with the fix now. It should be into and possibly through review tomorrow.

Show

Christopher Waters added a comment - 13/Oct/20 11:34 PM I'm testing the `ip_isr` branch with the fix now. It should be into and possibly through review tomorrow.

Hide

Permalink

Christopher Waters added a comment - 14/Oct/20 6:26 PM

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32851/pipeline

Show

Christopher Waters added a comment - 14/Oct/20 6:26 PM https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32851/pipeline

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 16/Oct/20 4:48 PM

I think this looks good; I just asked for some minor clarifications, especially in the part that seems to be doing the fix for this ticket (len(newRow)). As a more general and broad comment, it seems that the debug part is always interactive (i.e., it waits for the user to provide input before continuing). Is this correct? This is the impression that I get from other debug options in other tasks (e.g, in crosstalk). If we want to run a script over many detectors and with multiple tasks, for example, and we also want to look at the debug plots, would that be possible without the interaction with the user?

Show

Andrés Alejandro Plazas Malagón added a comment - 16/Oct/20 4:48 PM I think this looks good; I just asked for some minor clarifications, especially in the part that seems to be doing the fix for this ticket ( len(newRow) ). As a more general and broad comment, it seems that the debug part is always interactive (i.e., it waits for the user to provide input before continuing). Is this correct? This is the impression that I get from other debug options in other tasks (e.g, in crosstalk). If we want to run a script over many detectors and with multiple tasks, for example, and we also want to look at the debug plots, would that be possible without the interaction with the user?

Data Management

constructDark.py fails on LSSTCam with 'Image contains no Pixels'

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Jenkins