Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5078

PcaPsf can hit an assertion failure

    Details

    • Type: Bug
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_algorithms
    • Labels:
      None
    • Templates:
    • Story Points:
      1
    • Team:
      Data Release Production

      Description

      This is bad for multiple reasons:
      1. When multiprocessing, the assertion failure kills a single process, which prevents the final join of the multiple processes, so the job hangs forever.
      2. The failure is not logged.
      3. Hard assertions like this should only occur when we break the system integrity, which this does not (i.e., it's too big a hammer for the problem).

      pprice@tiger-sumire:/tigress/pprice/dm-4692 $ eups list -s
      afw                   tickets.DM-4692-gd8ad35cd96+1     b1901 setup
      afwdata               2016_01.0         b1901 b1902 setup
      astrometry_net        0.50.lsst2+5      b1901 b1902 setup
      astrometry_net_data   sdss-dr9-fink-v5b         setup
      base                  2016_01.0         b1901 b1902 setup
      boost                 1.59.lsst5        b1901 b1902 setup
      cfitsio               3360.lsst4        b1901 b1902 setup
      coadd_chisquared      2016_01.0+6       b1901 setup
      coadd_utils           2016_01.0+6       b1901 setup
      daf_base              2016_01.0         b1901 b1902 setup
      daf_butlerUtils       tickets.DM-4692-g048b33c50e+3     b1901 setup
      daf_persistence       2016_01.0-1-gf47bb69+1    b1901 b1902 setup
      display_ds9           2015_10.0+43      b1901 setup
      doxygen               1.8.5.lsst1       b1901 b1902 setup
      eigen                 3.2.5             b1901 b1902 setup
      fftw                  3.3.4.lsst2       b1901 b1902 setup
      geom                  10.0+50           b1901 b1902 setup
      gsl                   1.16.lsst3        b1901 b1902 setup
      ip_diffim             tickets.DM-4692-g543ea8fde5+3     b1901 setup
      ip_isr                2016_01.0+6       b1901 setup
      lsst_build            LOCAL:/tigress/pprice/lsstsw/lsst_build   setup
      mariadbclient         master-gf2dee38289        b1901 b1902 setup
      matplotlib            0.0.1+5           b1901 b1902 setup
      meas_algorithms       tickets.DM-4692-g3d073a93d7+1     b1901 setup
      meas_astrom           tickets.DM-4692-gbbf15418e6+1     b1901 setup
      meas_base             LOCAL:/tigress/pprice/dm-4692/meas_base   setup
      meas_deblender        2016_01.0+6       b1901 setup
      minuit2               5.28.00.lsst2     b1901 b1902 setup
      ndarray               10.1+58           b1901 b1902 setup
      numpy                 0.0.1+5           b1901 b1902 setup
      obs_subaru            LOCAL:/tigress/pprice/dm-4692/obs_subaru  setup
      obs_test              tickets.DM-4692-g1533aee20f+1     b1901 setup
      pex_config            2016_01.0         b1901 b1902 setup
      pex_exceptions        2016_01.0         b1901 b1902 setup
      pex_logging           2016_01.0         b1901 b1902 setup
      pex_policy            2016_01.0         b1901 b1902 setup
      pipe_base             2016_01.0+6       b1901 setup
      pipe_tasks            LOCAL:/tigress/pprice/dm-4692/pipe_tasks  setup
      psfex                 2016_01.0         b1901 b1902 setup
      pyfits                3.4.0             b1901 b1902 setup
      python                0.0.3             b1901 b1902 setup
      python_d2to1          0.2.12            b1901 b1902 setup
      pyyaml                3.11.lsst1        b1901 b1902 setup
      scons                 2.3.5             b1901 b1902 setup
      sconsUtils            2016_01.0         b1901 b1902 setup
      skymap                2016_01.0+6       b1901 setup
      skypix                10.0+347          b1901 setup
      stsci_distutils       0.3.7-1-gb22a065  b1901 b1902 setup
      swig                  3.0.2.lsst1       b1901 b1902 setup
      utils                 2016_01.0         b1901 b1902 setup
      wcslib                5.13.lsst1        b1901 b1902 setup
      xpa                   2.1.15.lsst3      b1901 b1902 setup
       
      pprice@tiger-sumire:/tigress/pprice/dm-4692 $ processCcd.py /tigress/HSC/HSC --rerun price/dm-4692 --rerun price/dm-4692 --id visit=1248 ccd=100 --clobber-config
      /tigress/pprice/lsstsw/miniconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
        warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
      : Loading config overrride file '/tigress/pprice/dm-4692/obs_subaru/config/processCcd.py'
      WARNING: Unable to use psfex: No module named extensions.psfex.psfexPsfDeterminer
      Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements
      Cannot enable shapeHSM (
        File "src/Utils.cc", line 42, in std::string lsst::utils::getPackageDir(const std::string&)
          Package meas_extensions_shapeHSM not found {0}
      lsst::pex::exceptions::NotFoundError: 'Package meas_extensions_shapeHSM not found'
      ): disabling HSM shape measurements
      : Loading config overrride file '/tigress/pprice/dm-4692/obs_subaru/config/hsc/processCcd.py'
      : input=/tigress/HSC/HSC
      : calib=None
      : output=/tigress/HSC/HSC/rerun/price/dm-4692
      CameraMapper: Loading registry registry from /tigress/HSC/HSC/rerun/price/dm-4692/_parent/registry.sqlite3
      CameraMapper: Loading calibRegistry registry from /tigress/HSC/HSC/CALIB/calibRegistry.sqlite3
      processCcd: Processing {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0}
      processCcd.isr: Performing ISR on sensor {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0}
      processCcd.isr: Applying linearity corrections to Ccd 100
      processCcd.isr.crosstalk: Applying crosstalk correction
      processCcd.isr: Set 0 BAD pixels to 3147.74
      processCcd.isr: Flattened sky level: 3847.800781 +/- 2114.507723
      processCcd.isr: Measuring sky levels in 8x16 grids: 3884.324645
      processCcd.isr: Sky flatness in 8x16 grids - pp: 15293.248379 rms: 1173.423587
      processCcd.isr: Setting rough magnitude zero point: 34.678409
      processCcd.charImage: Processing {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0}
      processCcd.charImage.repair: Identified 6044 cosmic rays.
      processCcd.charImage.detectAndMeasure.detection: Detected 127 positive sources to 5 sigma.
      processCcd.charImage.detectAndMeasure.detection: Resubtracting the background after object detection
      processCcd.charImage.detectAndMeasure.measurement: Measuring 127 sources (127 parents, 0 children) 
      processCcd.charImage.measurePsf: Measuring PSF
      /tigress/pprice/lsstsw/stack/Linux64/meas_algorithms/tickets.DM-4692-g3d073a93d7+1/python/lsst/meas/algorithms/objectSizeStarSelector.py:354: RuntimeWarning: invalid value encountered in less
        bad = numpy.logical_or(bad, width < self._widthMin)
      /tigress/pprice/lsstsw/stack/Linux64/meas_algorithms/tickets.DM-4692-g3d073a93d7+1/python/lsst/meas/algorithms/objectSizeStarSelector.py:355: RuntimeWarning: invalid value encountered in greater
        bad = numpy.logical_or(bad, width > self._widthMax)
      processCcd.charImage.measurePsf: PSF star selector found 6 candidates
      meas.algorithms.psfDeterminer WARNING: You only have 3 eigen images (you asked for 4): reducing number of eigen components
      meas.algorithms.psfDeterminer WARNING: You only have 1 eigen images (you asked for 3): reducing number of eigen components
      meas.algorithms.psfDeterminer WARNING: You only have 1 eigen images (you asked for 2): reducing number of eigen components
      python: /tigress/pprice/lsstsw/stack/Linux64/eigen/3.2.5/include/Eigen/src/Core/Redux.h:202: static Eigen::internal::redux_impl<Func, Derived, 3, 0>::Scalar Eigen::internal::redux_impl<Func, Derived, 3, 0>::run(const Derived&, const Func&) [with Func = Eigen::internal::scalar_max_op<double>; Derived = Eigen::CwiseUnaryOp<Eigen::internal::scalar_abs_op<double>, const Eigen::Matrix<double, -1, -1> >; Eigen::internal::redux_impl<Func, Derived, 3, 0>::Scalar = double]: Assertion `size && "you are using an empty matrix"' failed.
      Aborted
      

      Note:

      • This occurred while testing DM-4692. The LOCAL pipe_tasks and obs_subaru are on that ticket branch. The LOCAL meas_base is for the fix from DM-5050.
      • One root cause of the bad PSF modeling may be bad rotations in the application of the calibs (Lauren MacArthur is looking into that; don't know if there's a ticket number), but this should never happen regardless.

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  price Paul Price
                  Watchers:
                  Jim Bosch, John Swinbank, Lauren MacArthur, Paul Price
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel