Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7538

CModel is producing SingularTransformException

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_modelfit
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Sprint:
      DRP F16-4
    • Team:
      Data Release Production

      Description

      This is running on private data because I haven't yet found a public example that produces the problem. The particular data set may be specialised, but we shouldn't be so chatty in any case.

      pprice@tiger-sumire:/scratch/gpfs/pprice $ processCcd.py DATA --rerun cmodel --id visit=42742 ccd=49 --clobber-config
      /home/pprice/.local/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
        warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
      : Loading config overrride file '/tigress/HSC/LSST/stack/Linux64/obs_subaru/12.0-16-gbc41bb6+2/config/processCcd.py'
      : Loading config overrride file '/tigress/HSC/LSST/stack/Linux64/obs_subaru/12.0-16-gbc41bb6+2/config/hsc/processCcd.py'
      : input=/scratch/gpfs/pprice/DATA
      : calib=None
      : output=/scratch/gpfs/pprice/DATA/rerun/cmodel
      CameraMapper: Loading registry registry from /scratch/gpfs/pprice/DATA/rerun/cmodel/_parent/registry.sqlite3
      CameraMapper: Loading calibRegistry registry from /scratch/gpfs/pprice/DATA/CALIB/calibRegistry.sqlite3
      processCcd: Processing {'taiObs': '2015-10-15', 'pointing': 1383, 'visit': 42742, 'dateObs': '2015-10-15', 'filter': 'HSC-I', 'field': 'U105A_CEN', 'ccd': 49, 'expTime': 250.0}
      processCcd.isr: Performing ISR on sensor {'taiObs': '2015-10-15', 'pointing': 1383, 'visit': 42742, 'dateObs': '2015-10-15', 'filter': 'HSC-I', 'field': 'U105A_CEN', 'ccd': 49, 'expTime': 250.0}
      processCcd.isr.crosstalk: Applying crosstalk correction
      processCcd.isr: Applying brighter fatter correction
      processCcd.isr: Finished brighter fatter in 6 iterations
      processCcd.isr: Set 0 BAD pixels to 3199.49
      processCcd.isr: Flattened sky level: 3200.822754 +/- 38.878743
      processCcd.isr: Measuring sky levels in 8x16 grids: 3196.589865
      processCcd.isr: Sky flatness in 8x16 grids - pp: 0.151048 rms: 0.022820
      processCcd.isr: Setting rough magnitude zero point: 34.594850
      processCcd.charImage: Processing {'taiObs': '2015-10-15', 'pointing': 1383, 'visit': 42742, 'dateObs': '2015-10-15', 'filter': 'HSC-I', 'field': 'U105A_CEN', 'ccd': 49, 'expTime': 250.0}
      processCcd.charImage.repair: Identified 831 cosmic rays.
      processCcd.charImage.detection: Detected 664 positive sources to 50 sigma.
      processCcd.charImage.detection: Resubtracting the background after object detection
      processCcd.charImage.measurement: Measuring 664 sources (664 parents, 0 children) 
      processCcd.charImage.measurePsf: Measuring PSF
      processCcd.charImage.measurePsf: PSF star selector found 303 candidates
      processCcd.charImage.measurePsf: Reserved 75 candidates from the fitting
      processCcd.charImage.measurePsf: PSF determination using 205/303 stars.
      processCcd.charImage: iter 1; PSF sigma=1.26, dimensions=(41, 41); median background=3196.07
      processCcd.charImage.repair: Identified 799 cosmic rays.
      processCcd.charImage.detection: Detected 743 positive sources to 50 sigma.
      processCcd.charImage.detection: Resubtracting the background after object detection
      processCcd.charImage.measurement: Measuring 743 sources (743 parents, 0 children) 
      processCcd.charImage.measurement WARNING: Error in modelfit_CModel.measure on record 36715308886523905: 
        File "src/geom/LinearTransform.cc", line 66, in const lsst::afw::geom::LinearTransform lsst::afw::geom::LinearTransform::invert() const
          Could not compute LinearTransform inverse {0}
      lsst::afw::geom::SingularTransformException: 'Could not compute LinearTransform inverse'
       
      processCcd.charImage.measurement WARNING: Error in modelfit_CModel.measure on record 36715308886523906: 
        File "src/geom/LinearTransform.cc", line 66, in const lsst::afw::geom::LinearTransform lsst::afw::geom::LinearTransform::invert() const
          Could not compute LinearTransform inverse {0}
      lsst::afw::geom::SingularTransformException: 'Could not compute LinearTransform inverse'
       
      [many, many more follow...]
      processCcd.charImage.measureApCorr: Measuring aperture corrections for 7 flux fields
      /tigress/HSC/LSST/stack/Linux64/meas_algorithms/12.0-24-g3f92e4c+6/python/lsst/meas/algorithms/measureApCorr.py:226: RuntimeWarning: invalid value encountered in greater
        fluxes > 0.0,
      processCcd.charImage.measureApCorr: Aperture correction for modelfit_CModel_exp: RMS 0.073826 from 253
      processCcd.charImage.measureApCorr: Aperture correction for base_PsfFlux: RMS 0.074407 from 255
      processCcd.charImage.measureApCorr: Aperture correction for modelfit_CModel: RMS 0.073742 from 253
      processCcd.charImage.measureApCorr: Aperture correction for modelfit_CModel_initial: RMS 0.072894 from 254
      processCcd.charImage.measureApCorr: Aperture correction for ext_photometryKron_KronFlux: RMS 0.062862 from 255
      processCcd.charImage.measureApCorr: Aperture correction for base_GaussianFlux: RMS 0.086553 from 255
      processCcd.charImage.measureApCorr: Aperture correction for modelfit_CModel_dev: RMS 0.073533 from 253
      processCcd.charImage.applyApCorr: Applying aperture corrections to 7 flux fields
      processCcd.charImage.applyApCorr: Use naive flux sigma computation
      processCcd.calibrate: Processing {'taiObs': '2015-10-15', 'pointing': 1383, 'visit': 42742, 'dateObs': '2015-10-15', 'filter': 'HSC-I', 'field': 'U105A_CEN', 'ccd': 49, 'expTime': 250.0}
      processCcd.calibrate.detection: Detected 4386 positive sources to 5 sigma.
      processCcd.calibrate.detection: Resubtracting the background after object detection
      processCcd.calibrate.deblend: Deblending 4386 sources
      processCcd.calibrate.deblend: Deblended: of 4386 sources, 1006 were deblended, creating 3754 children, total 8140 sources
      processCcd.calibrate.measurement: Measuring 8140 sources (4386 parents, 3754 children) 
      processCcd.calibrate.applyApCorr: Applying aperture corrections to 2 flux fields
      processCcd.calibrate.applyApCorr: Use naive flux sigma computation
      [...]
      

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          I'll have a go at fixing the problem, and hand it over if it's not easy.

          Show
          price Paul Price added a comment - I'll have a go at fixing the problem, and hand it over if it's not easy.
          Hide
          price Paul Price added a comment -

          #0  0x00002aaab33ffca0 in __cxa_throw () from /usr/lib64/libstdc++.so.6
          #1  0x00002aaadaf172b6 in lsst::afw::geom::LinearTransform::invert (
              this=this@entry=0x3007778) at src/geom/LinearTransform.cc:63
          #2  0x00002aab08ae8620 in lsst::shapelet::ShapeletFunctionEvaluator::_computeRawMoments (this=this@entry=0x3007760, q0=@0x7ffffffec968: 0, q1=..., q2=...)
              at src/ShapeletFunction.cc:213
          #3  0x00002aab08b14b93 in lsst::shapelet::MultiShapeletFunctionEvaluator::computeMoments (this=this@entry=0x7ffffffecfc0) at src/MultiShapeletFunction.cc:136
          #4  0x00002aab087e9991 in lsst::meas::modelfit::CModelAlgorithm::_applyImpl (
              this=this@entry=0x33ff0b0, result=..., exposure=..., psf=..., center=..., 
              moments=..., approxFlux=approxFlux@entry=237964.85741607359, 
              kronRadius=2.027911901473999, footprintArea=361) at src/CModel.cc:1061
          #5  0x00002aab087eaeb9 in lsst::meas::modelfit::CModelAlgorithm::measure (
              this=this@entry=0x33ff0b0, measRecord=..., exposure=...)
              at src/CModel.cc:1352
          #6  0x00002aab0810059e in _wrap_CModelAlgorithm_measure__SWIG_0 (
              args=args@entry=0x2aab09910b90)
              at python/lsst/meas/modelfit/modelfitLib_wrap.cc:80496
          

          Show
          price Paul Price added a comment - #0 0x00002aaab33ffca0 in __cxa_throw () from /usr/lib64/libstdc++.so.6 #1 0x00002aaadaf172b6 in lsst::afw::geom::LinearTransform::invert ( this=this@entry=0x3007778) at src/geom/LinearTransform.cc:63 #2 0x00002aab08ae8620 in lsst::shapelet::ShapeletFunctionEvaluator::_computeRawMoments (this=this@entry=0x3007760, q0=@0x7ffffffec968: 0, q1=..., q2=...) at src/ShapeletFunction.cc:213 #3 0x00002aab08b14b93 in lsst::shapelet::MultiShapeletFunctionEvaluator::computeMoments (this=this@entry=0x7ffffffecfc0) at src/MultiShapeletFunction.cc:136 #4 0x00002aab087e9991 in lsst::meas::modelfit::CModelAlgorithm::_applyImpl ( this=this@entry=0x33ff0b0, result=..., exposure=..., psf=..., center=..., moments=..., approxFlux=approxFlux@entry=237964.85741607359, kronRadius=2.027911901473999, footprintArea=361) at src/CModel.cc:1061 #5 0x00002aab087eaeb9 in lsst::meas::modelfit::CModelAlgorithm::measure ( this=this@entry=0x33ff0b0, measRecord=..., exposure=...) at src/CModel.cc:1352 #6 0x00002aab0810059e in _wrap_CModelAlgorithm_measure__SWIG_0 ( args=args@entry=0x2aab09910b90) at python/lsst/meas/modelfit/modelfitLib_wrap.cc:80496
          Hide
          price Paul Price added a comment -

          This will block the chattiness. The shapelet PSF appears to have NANs in it. I didn't see any simple way of detecting that before the fact, so went with snuffing the exception with a MeasurementError. Jim Bosch, please let me know if you want to go a different way.

          pprice@tiger-sumire:/scratch/gpfs/pprice/meas_modelfit (tickets/DM-7538=) $ git sub-patch
          commit 687051dcc0766a9cb5e06fd22112f7bbca771ec4
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Tue Sep 6 22:55:07 2016 -0400
           
              trap SingularTransformException to reduce chatter
           
          diff --git a/src/CModel.cc b/src/CModel.cc
          index dd3cc8f..0769888 100644
          --- a/src/CModel.cc
          +++ b/src/CModel.cc
          @@ -1058,7 +1058,16 @@ void CModelAlgorithm::_applyImpl(
               int footprintArea
           ) const {
           
          -    afw::geom::ellipses::Quadrupole psfMoments = psf.evaluate().computeMoments().getCore();
          +    afw::geom::ellipses::Quadrupole psfMoments;
          +    try {
          +        psfMoments = psf.evaluate().computeMoments().getCore();
          +    } catch (afw::geom::SingularTransformException const& exc) {
          +        throw LSST_EXCEPT(
          +            meas::base::MeasurementError,
          +            std::string("Singular transform in shapelets: ") + exc.what(),
          +            CModelResult::NO_SHAPELET_PSF
          +            );
          +    }
           
               PixelFitRegion region(getControl().region, moments, psfMoments, kronRadius, footprintArea);
               result.initialFitRegion = region.ellipse;
          @@ -1325,7 +1334,15 @@ void CModelAlgorithm::measure(
                   (measRecord.getTable()->getShapeFlagKey().isValid() && measRecord.getShapeFlag())) {
                   if (getControl().fallbackInitialMomentsPsfFactor > 0.0) {
                       result.setFlag(Result::NO_SHAPE, true);
          -            moments = psf.evaluate().computeMoments().getCore();
          +            try {
          +                moments = psf.evaluate().computeMoments().getCore();
          +            } catch (afw::geom::SingularTransformException const& exc) {
          +                throw LSST_EXCEPT(
          +                    meas::base::MeasurementError,
          +                    std::string("Singular transform in shapelets: ") + exc.what(),
          +                    CModelResult::NO_SHAPELET_PSF
          +                );
          +            }
                       moments.scale(getControl().fallbackInitialMomentsPsfFactor);
                   } else {
                       throw LSST_EXCEPT(
          

          Show
          price Paul Price added a comment - This will block the chattiness. The shapelet PSF appears to have NANs in it. I didn't see any simple way of detecting that before the fact, so went with snuffing the exception with a MeasurementError . Jim Bosch , please let me know if you want to go a different way. pprice@tiger-sumire:/scratch/gpfs/pprice/meas_modelfit (tickets/DM-7538=) $ git sub-patch commit 687051dcc0766a9cb5e06fd22112f7bbca771ec4 Author: Paul Price <price@astro.princeton.edu> Date: Tue Sep 6 22:55:07 2016 -0400   trap SingularTransformException to reduce chatter   diff --git a/src/CModel.cc b/src/CModel.cc index dd3cc8f..0769888 100644 --- a/src/CModel.cc +++ b/src/CModel.cc @@ -1058,7 +1058,16 @@ void CModelAlgorithm::_applyImpl( int footprintArea ) const { - afw::geom::ellipses::Quadrupole psfMoments = psf.evaluate().computeMoments().getCore(); + afw::geom::ellipses::Quadrupole psfMoments; + try { + psfMoments = psf.evaluate().computeMoments().getCore(); + } catch (afw::geom::SingularTransformException const& exc) { + throw LSST_EXCEPT( + meas::base::MeasurementError, + std::string("Singular transform in shapelets: ") + exc.what(), + CModelResult::NO_SHAPELET_PSF + ); + } PixelFitRegion region(getControl().region, moments, psfMoments, kronRadius, footprintArea); result.initialFitRegion = region.ellipse; @@ -1325,7 +1334,15 @@ void CModelAlgorithm::measure( (measRecord.getTable()->getShapeFlagKey().isValid() && measRecord.getShapeFlag())) { if (getControl().fallbackInitialMomentsPsfFactor > 0.0) { result.setFlag(Result::NO_SHAPE, true); - moments = psf.evaluate().computeMoments().getCore(); + try { + moments = psf.evaluate().computeMoments().getCore(); + } catch (afw::geom::SingularTransformException const& exc) { + throw LSST_EXCEPT( + meas::base::MeasurementError, + std::string("Singular transform in shapelets: ") + exc.what(), + CModelResult::NO_SHAPELET_PSF + ); + } moments.scale(getControl().fallbackInitialMomentsPsfFactor); } else { throw LSST_EXCEPT(
          Hide
          jbosch Jim Bosch added a comment -

          Looks fine. I'd ultimately like to have CModel also check for flags in the PSF approximation algorithm - and it'd be very good to know if those are at least being set in this case - but that would require more structural changes, and this is a good defensive addition even if we were to add that.

          Show
          jbosch Jim Bosch added a comment - Looks fine. I'd ultimately like to have CModel also check for flags in the PSF approximation algorithm - and it'd be very good to know if those are at least being set in this case - but that would require more structural changes, and this is a good defensive addition even if we were to add that.
          Hide
          price Paul Price added a comment -

          Thanks Jim.

          Merged to master after passing Jenkins.

          Show
          price Paul Price added a comment - Thanks Jim. Merged to master after passing Jenkins .

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: