John Swinbank added a comment - 06/Jan/15 12:57 PM Lauren MacArthur , Jim Bosch , Robert Lupton and John Swinbank discussed this issue on 2015-01-06. The basic idea here is to re-enable the measurement algorithms which were disabled during the work on DM-245 . (Most of?) these are currently being transition from meas_algorithms to meas_base on the LSST side. The aim here is to enable the new measurement framework and associated plugins working with the HSC data. The scope for this issue is to perform basic sanity checking about this – to ensure that the code runs and doesn't do anything obviously insane – not to perform science quality validation.

Lauren MacArthur added a comment - 30/Jan/15 3:11 PM Notes for DM-1573 The measurement algorithms that could NOT be added back in are: root.measurement.algorithms.names |= ["jacobian", "focalplane"] because these don't exist in LSST. processCcd.py was run for all CCDs of a COSMOS field: pointing 817, visit 1228, filter HSC-I. 52 CCDs ran all the way though (BLUE on figure) 46 failed the astrometry solution (RED on figure: the pattern suggests distortion correction issues): processCcd.calibrate.astrometry WARNING: Did not get an astrometric solution from Astrometry.net processCcd.calibrate.astrometry: Searching around RA,Dec = (150.215,1.49597) with radius 1 deg yields 26871 reference-catalog sources processCcd FATAL: Failed on dataId={'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1228, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 1, 'expTime': 270.0}: Unable to match sources with catalog. RuntimeError: Unable to match sources with catalog. 2 failed with ( Note request to report as a bug ) (GREEN on figure): processCcd FATAL: Failed on dataId={'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1228, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 7, 'expTime': 270.0}: The algorithm 'CircularApertureFluxSingleFramePlugin' thinks it cannot fail, but it did; please report this as a bug (the full traceback is above). Traceback (most recent call last): File "/tigress/HSC/LSST/lsstsw/stack/Linux64/pipe_base/master-g8a4579b8a5+3/python/lsst/pipe/base/cmdLineTask.py", line 315, in __call__ result = task.run(dataRef, **kwargs) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/pipe_base/master-g8a4579b8a5+3/python/lsst/pipe/base/timer.py", line 118, in wrapper res = func(self, *args, **keyArgs) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/pipe_tasks/master-g379087737a+36/python/lsst/pipe/tasks/processCcd.py", line 82, in run result = self.process(sensorRef, postIsrExposure) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/pipe_base/master-g8a4579b8a5+3/python/lsst/pipe/base/timer.py", line 118, in wrapper res = func(self, *args, **keyArgs) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/pipe_tasks/master-g379087737a+36/python/lsst/pipe/tasks/processImage.py", line 190, in process self.measurement.run(calExposure, sources) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/meas_base/master-g37f9e1d64b+5/python/lsst/meas/base/sfm.py", line 394, in run self.callMeasure(measParentRecord, exposure) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/meas_base/master-g37f9e1d64b+5/python/lsst/meas/base/base.py", line 420, in callMeasure plugin.fail(measRecord) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/meas_base/master-g37f9e1d64b+5/python/lsst/meas/base/base.py", line 265, in fail raise NotImplementedError(message) NotImplementedError: The algorithm 'CircularApertureFluxSingleFramePlugin' thinks it cannot fail, but it did; please report this as a bug (the full traceback is above). Other warnings of note: Runtime Warning (most CCDs that get all the way through processCcd.py): ========================================================= processCcd.calibrate.measurePsf: Measuring PSF /tigress/HSC/LSST/lsstsw/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice. warnings.warn("Mean of empty slice.", RuntimeWarning) /tigress/HSC/LSST/lsstsw/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:71: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) /tigress/HSC/LSST/lsstsw/stack/Linux64/meas_algorithms/master-g7bce1f4e06+28/python/lsst/meas/algorithms/objectSizeStarSelector.py:144: RuntimeWarning: invalid value encountered in less update = dist < minDist CCD 18 (warnings, still made it through): ================================= meas.algorithms.objectSizeStarSelector: Failed to make a psfCandidate from source 1054921277309275: File "src/image/Image.cc", line 88, in static typename lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const lsst::afw::geom::Extent2I&, const lsst::afw::geom::Extent2I&, const typename lsst::afw::image::detail::types_traits<PixelT, false>::view_t&) [with PixelT = float] Box2I(Point2I(2029,4122),Extent2I(21,21)) doesn't fit in image 2048x4176 {0} File "src/PsfCandidate.cc", line 200, in boost::shared_ptr<lsst::afw::image::MaskedImage<EntryT, short unsigned int, float> > lsst::meas::algorithms::PsfCandidate<PixelT>::extractImage(unsigned int, unsigned int) const [with PixelT = float] Extracting image of PSF candidate {1} lsst::pex::exceptions::LengthError: 'Box2I(Point2I(2029,4122),Extent2I(21,21)) doesn't fit in image 2048x4176 {0}; Extracting image of PSF candidate {1}' CCD 19, 82 (warnings, still made it through): ==================================== processCcd.deblend: Deblending 798 sources lsst.meas.deblender.symmetrizeFootprint WARNING: Failed to find span containing (1813,2471): nearest is 2388, [1875,1875]. Footprint bbox is [1823,1882],[2291,2388] processCcd.deblend WARNING: Error deblending source 1054925572277004: 'NoneType' object has no attribute 'clipTo' Traceback (most recent call last): File "/tigress/HSC/LSST/lsstsw/stack/Linux64/meas_deblender/master-gb769b6c842+52/python/lsst/meas/deblender/deblend.py", line 280, in deblend clipStrayFluxFraction=self.config.clipStrayFluxFraction, File "/tigress/HSC/LSST/lsstsw/stack/Linux64/meas_deblender/master-gb769b6c842+52/python/lsst/meas/deblender/baseline.py", line 355, in deblend psf, pk, sigma1, patchEdges) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/meas_deblender/master-gb769b6c842+52/python/lsst/meas/deblender/baseline.py", line 1102, in _handle_flux_at_edge tfoot2.clipTo(imbb) AttributeError: 'NoneType' object has no attribute 'clipTo' CCD 36, 58, 79, 93 (warnings, still made it through): ======================================== meas.algorithms.objectSizeStarSelector: Failed to make a psfCandidate from source 1054998586720393: File "src/image/Image.cc", line 88, in static typename lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const lsst::afw::geom::Extent2I&, const lsst::afw::geom::Extent2I&, const typename lsst::afw::image::detail::types_traits<PixelT, false>::view_t&) [with PixelT = float] Box2I(Point2I(-2,2553),Extent2I(21,21)) doesn't fit in image 2048x4176 {0} File "src/PsfCandidate.cc", line 200, in boost::shared_ptr<lsst::afw::image::MaskedImage<EntryT, short unsigned int, float> > lsst::meas::algorithms::PsfCandidate<PixelT>::extractImage(unsigned int, unsigned int) const [with PixelT = float] Extracting image of PSF candidate {1} lsst::pex::exceptions::LengthError: 'Box2I(Point2I(-2,2553),Extent2I(21,21)) doesn't fit in image 2048x4176 {0}; Extracting image of PSF candidate {1}' CCD 7 (warning) ======================================== processCcd.measurement: Measuring 1390 sources (969 parents, 421 children) processCcd.measurement WARNING: Error in base_PixelFlags.measure on record 1054874032668992: File "src/PixelFlags.cc", line 82, in static void lsst::meas::base::PixelFlagsAlgorithm::apply(const lsst::afw::image::MaskedImage<T, short unsigned int, float>&, const lsst::afw::geom::Point2D&, const lsst::afw::detection::Footprint&, lsst::meas::base::PixelFlagsAlgorithm::Result&, const lsst::meas::base::PixelFlagsAlgorithm::Control&) [with T = float] Center point passed to PixelFlagsALgorithm is NaN {0} lsst::pex::exceptions::InvalidParameterError: 'Center point passed to PixelFlagsALgorithm is NaN' The above figures compare the CORR images for a given CCD (top) and the full array (loaded with -mosaic in ds9) from an HSC-stack rerun (middle) vs. the LSST-stack rerun (bottom). It seems many (most!) of the calibration stages are not working properly. These were not expected to work out of-the-box on the LSST side and will need to be addressed in a separate issue.

Lauren MacArthur added a comment - 26/Feb/15 3:18 PM - edited The calibrated issues noted in CORR-0001228-050_comp.png have been resolved as part of DM-245 . The calibration frames for the rotated CCDs in the HSC camera were not being rotated back before being applied. Astrometric solutions are now succeeding for many more of the CCDs around the edge that previously failed, as shown in FocalPlane_0001228a.png. The strange arrangement in the CCDs seen in FocalPlane_0001228a.png seems to be an issue with ds9 (getting confused with the wcs?) rather than an erroneous astrometric solution as individual frames loaded into ds9 line up nicely with DSS images and Sloan object catalogues. Indeed, if I only load in the CCDs 80-99, they do line up reasonably well. Also, I ran the same visit on the HSC stack having turned off hscAstrom and using LSST's meas_astrom instead and the same pathology is observed when loading the CORR*.fits images into ds9 -mosaic. HSC's distEst package has been merged in to obs_subaru ( Paul Price 's DM-1794 branch). Things are running, but there are some issues to be solved in regards to what pixel units and scalings need to go where while keeping the camera.py entries consistent with those in all other obs_XXX (distest works in units of pixels rather than mm). This will be part of a separate ticket: DM-2194 .