Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5758

processEimage.py raises various errors in .calibrate.measureApCorr

    Details

    • Templates:

      Description

      Running on phosim data for Twinkles, we're seeing ZeroDivisionErrors and InvalidParameterErrors for ~1% of the eimage data.

      I've placed example data in /nfs/home/jchiang/Twinkles at lsst-dev.

      Here is the command and traceback for the ZeroDivisionError:

      [obs_lsstSim b1953] processEimage.py input --id visit=1697655 --output output --doraise --clobber-config
       
      < skipped output>
       
      processEimage.calibrate.measureApCorr: Measuring aperture corrections for 2 flux fields
      processEimage.calibrate.measureApCorr: Aperture correction for base_PsfFlux: RMS 0.063219 from 143
      Traceback (most recent call last):
        File "/home/lsstsw/stack/Linux64/obs_lsstSim/2016_01.0-2-ga2b3e68+2/bin/processEimage.py", line 25, in <module>
          ProcessEimageTask.parseAndRun()
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/cmdLineTask.py", line 444, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/cmdLineTask.py", line 199, in run
          resultList = mapFunc(self, targetList)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/cmdLineTask.py", line 318, in __call__
          result = task.run(dataRef, **kwargs)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/obs_lsstSim/2016_01.0-2-ga2b3e68+2/python/lsst/obs/lsstSim/processEimage.py", line 123, in run
          result = ProcessCcdTask.run(self, sensorRef)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/pipe_tasks/2016_01.0-11-g81670f4+2/python/lsst/pipe/tasks/processCcd.py", line 157, in run
          calib = self.calibrate.run(postIsrExposure, idFactory=idFactory, expId=expId)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/pipe_tasks/2016_01.0-11-g81670f4+2/python/lsst/pipe/tasks/calibrate.py", line 552, in run
          apCorrMap = self.measureApCorr.run(bbox=exposure.getBBox(), catalog=sources).apCorrMap
        File "/home/lsstsw/stack/Linux64/meas_base/2016_01.0-9-g0a48b21+5/python/lsst/meas/base/measureApCorr.py", line 198, in run
          apCorrData[n] = record.get(self.refFluxKeys.flux)/record.get(keys.flux)
      ZeroDivisionError: float division by zero
      [obs_lsstSim b1953] 
      

      Here is the same for the InvalidParameterError:

      [obs_lsstSim b1953] processEimage.py input --id visit=712850 --output output --doraise --clobber-config
       
      <skip output>
       
      processEimage.calibrate.measureApCorr: Aperture correction for base_PsfFlux: RMS 0.033659 from 154
      /home/lsstsw/stack/Linux64/meas_base/2016_01.0-9-g0a48b21+5/python/lsst/meas/base/measureApCorr.py:213: RuntimeWarning: invalid value encountered in less_equal
        keep = numpy.fabs(apCorrDiffs) <= apCorrDiffLim
      Traceback (most recent call last):
        File "/home/lsstsw/stack/Linux64/obs_lsstSim/2016_01.0-2-ga2b3e68+2/bin/processEimage.py", line 25, in <module>
          ProcessEimageTask.parseAndRun()
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/cmdLineTask.py", line 444, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/cmdLineTask.py", line 199, in run
          resultList = mapFunc(self, targetList)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/cmdLineTask.py", line 318, in __call__
          result = task.run(dataRef, **kwargs)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/obs_lsstSim/2016_01.0-2-ga2b3e68+2/python/lsst/obs/lsstSim/processEimage.py", line 123, in run
          result = ProcessCcdTask.run(self, sensorRef)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/pipe_tasks/2016_01.0-11-g81670f4+2/python/lsst/pipe/tasks/processCcd.py", line 157, in run
          calib = self.calibrate.run(postIsrExposure, idFactory=idFactory, expId=expId)
        File "/home/lsstsw/stack/Linux64/pipe_base/2016_01.0-2-g1dce45a+2/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/pipe_tasks/2016_01.0-11-g81670f4+2/python/lsst/pipe/tasks/calibrate.py", line 552, in run
          apCorrMap = self.measureApCorr.run(bbox=exposure.getBBox(), catalog=sources).apCorrMap
        File "/home/lsstsw/stack/Linux64/meas_base/2016_01.0-9-g0a48b21+5/python/lsst/meas/base/measureApCorr.py", line 203, in run
          apCorrField = ChebyshevBoundedField.fit(bbox, x, y, apCorrData, ctrl)
        File "/home/lsstsw/stack/Linux64/afw/2.2016.10-1-g690005f+1/python/lsst/afw/math/mathLib.py", line 9089, in fit
          return _mathLib.ChebyshevBoundedField_fit(*args)
      lsst.pex.exceptions.wrappers.InvalidParameterError: 
        File "src/math/LeastSquares.cc", line 434, in void lsst::afw::math::LeastSquares::_factor(bool)
          Number of columns of design matrix (6) must be smaller than number of data points (0) {0}
      lsst::pex::exceptions::InvalidParameterError: 'Number of columns of design matrix (6) must be smaller than number of data points (0)'
       
      [obs_lsstSim b1953] 
      

      We see these errors for tags b1936 through b1953. Later tags fail immediately with

      AttributeError: 'CalibrateConfig' object has no attribute 'repair'
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                krughoff Simon Krughoff
                Reporter:
                jchiang James Chiang
                Watchers:
                James Chiang, Paul Price, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel