Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27157

constructDark.py fails on LSSTCam with 'Image contains no Pixels'

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_drivers
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      I ran a PTC task on BOT run 12606, the first time we have the full LSSTCam.  Six CCDs failed to return results.  I am tracking down the reasons.  CCD R01_S00, detector=0 failed because it failed to create a master dark with the error "Image contains no Pixels". I have copied a command line that will reproduce the error, as well as the error itself below.  I also have a screenshot of the DS9 image of the input dark.  It contains a large bright defect, which may be the cause of the error.  I tried several other 300 second darks, including 3020100800105, but they gave the same error.

      Command string:

      constructDark.py /project/shared/BOT --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12606/CALIB --rerun /project/shared/BOT/rerun/cslage/PTC_Test  --batch-type=smp  --cores 1  -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1  --clobber-config --id expId=3020100800045 detector=0
      

      Error:

      dark WARN: Unable to process DataId(initialdata={'expId': 3020100800045, 'detector': 0, 'dayObs': '2020-10-08', 'raftName': 'R01', 'detectorName': 'S00', 'snap': 0}, tag=set()): 
        File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter<short unsigned int>; MaskT = lsst::afw::math::MaskImposter<int>; VarianceT = lsst::afw::math::MaskImposter<float>; WeightT = lsst::afw::math::MaskImposter<float>]
          Image contains no pixels {0}
      lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'InvalidParameterError on lsst-devl01:2444703 in reduce: 
        File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter<short unsigned int>; MaskT = lsst::afw::math::MaskImposter<int>; VarianceT = lsst::afw::math::MaskImposter<float>; WeightT = lsst::afw::math::MaskImposter<float>]
          Image contains no pixels {0}
      lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'Traceback (most recent call last):
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 112, in wrapper
          return func(*args, **kwargs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 239, in wrapper
          return func(*args, **kwargs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 715, in reduce
          *args, **kwargs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 572, in _reduceQueue
          resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/20.0.0+2e34101df1/python/lsst/ctrl/pool/pool.py", line 572, in <listcomp>
          resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 625, in process
          exposure = self.processSingle(sensorRef, **kwargs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 983, in processSingle
          exposure = CalibTask.processSingle(self, sensorRef)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/20.0.0-1-g5b95a8c+d3005c1e37/python/lsst/pipe/drivers/constructCalibs.py", line 644, in processSingle
          return self.isr.runDataRef(dataRef).exposure
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef
          result = self.run(ccdExposure, camera=camera, **isrData.getDict())
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/20.0.0-19-gcdd82e7+6f5ab6e0f6/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 1366, in run
          overscanResults = self.overscanCorrection(ccdExposure, amp)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/isrTask.py", line 2008, in overscanCorrection
          overscanResults = self.overscan.run(ampImage.getImage(), overscanImage)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 140, in run
          overscanResult = self.measureVectorOverscan(overscanImage)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 459, in measureVectorOverscan
          overscanVector = self.collapseArrayMedian(masked)
        File "/software/lsstsw/stack_20200922/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/20.0.0-18-g206d22f+d2859fcb1f/python/lsst/ip/isr/overscan.py", line 345, in collapseArrayMedian
          rowMedian = afwMath.makeStatistics(newRow, fitType, self.statControl).getValue()
      lsst.pex.exceptions.wrappers.InvalidParameterError: 
        File "src/math/Statistics.cc", line 855, in void lsst::afw::math::Statistics::doStatistics(const ImageT&, const MaskT&, const VarianceT&, const WeightT&, int, const lsst::afw::math::StatisticsControl&) [with ImageT = lsst::afw::math::ImageImposter<short unsigned int>; MaskT = lsst::afw::math::MaskImposter<int>; VarianceT = lsst::afw::math::MaskImposter<float>; WeightT = lsst::afw::math::MaskImposter<float>]
          Image contains no pixels {0}
      lsst::pex::exceptions::InvalidParameterError: 'Image contains no pixels'
      

        Attachments

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment -

            This appears to be a failure mode of the overscan code.  Outliers are found by calculating statistics over the entire overscan region, with discrepant points masked then.  For this camera, the overscan is then collapsed along the serial direction by taking the MEDIAN_PER_ROW.  If all the points in that row have been masked, then the statistic raises. For the image in the example, this was occuring on amp C14.

            An updated ip_isr branch catches this failure, and sets the overscan to NaN (which will result in masking downstream, but no other value makes sense here).

            Show
            czw Christopher Waters added a comment - This appears to be a failure mode of the overscan code.  Outliers are found by calculating statistics over the entire overscan region, with discrepant points masked then.  For this camera, the overscan is then collapsed along the serial direction by taking the MEDIAN_PER_ROW.  If all the points in that row have been masked, then the statistic raises. For the image in the example, this was occuring on amp C14. An updated ip_isr branch catches this failure, and sets the overscan to NaN (which will result in masking downstream, but no other value makes sense here).
            Hide
            cslage Craig Lage added a comment -

            Yes, it runs with isr.overscan.fitType=MEDIAN instead of MEDIAN_PER_ROW.  I'll run this way for now.

            Show
            cslage Craig Lage added a comment - Yes, it runs with isr.overscan.fitType=MEDIAN instead of MEDIAN_PER_ROW.  I'll run this way for now.
            Hide
            cslage Craig Lage added a comment -

            I accidentally blew away that repo, so that CALIB directory is gone.  I thought clobber-ouput only deleted what was being changed, but it deletes everything!.  I'm re-building the repo now, but it will take a while.

            Show
            cslage Craig Lage added a comment - I accidentally blew away that repo, so that CALIB directory is gone.  I thought clobber-ouput only deleted what was being changed, but it deletes everything!.  I'm re-building the repo now, but it will take a while.
            Hide
            czw Christopher Waters added a comment -

            I'm testing the `ip_isr` branch with the fix now.  It should be into and possibly through review tomorrow.

            Show
            czw Christopher Waters added a comment - I'm testing the `ip_isr` branch with the fix now.  It should be into and possibly through review tomorrow.
            Show
            czw Christopher Waters added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32851/pipeline
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            I think this looks good; I just asked for some minor clarifications, especially in the part that seems to be doing the fix for this ticket (len(newRow)). As a more general and broad comment, it seems that the debug part is always interactive (i.e., it waits for the user to provide input before continuing). Is this correct? This is the impression that I get from other debug options in other tasks (e.g, in crosstalk). If we want to run a script over many detectors and with multiple tasks, for example, and we also want to look at the debug plots, would that be possible without the interaction with the user?

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - I think this looks good; I just asked for some minor clarifications, especially in the part that seems to be doing the fix for this ticket ( len(newRow) ). As a more general and broad comment, it seems that the debug part is always interactive (i.e., it waits for the user to provide input before continuing). Is this correct? This is the impression that I get from other debug options in other tasks (e.g, in crosstalk). If we want to run a script over many detectors and with multiple tasks, for example, and we also want to look at the debug plots, would that be possible without the interaction with the user?

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              cslage Craig Lage
              Reviewers:
              Andrés Alejandro Plazas Malagón
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.