# Investigate single frame processing astrometry failures/poor solutions on some HSC chip/visits.

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
6
• Sprint:
DRP F16-2
• Team:
Data Release Production

## Description

The astrometric solution of some visit/ccd combinations for HSC data are failing or finding very poor solutions. This typically occurs for the outermost (highly fringed) ccds (e.g. 100..103, 95). I provide some sample output below.

 LSST bad fit: visit=19696 ccd=100 processCcd.calibrate.astrometry.refObjLoader: Loaded 71 reference objects processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 71 stars processCcd.calibrate.astrometry.matcher: Purged 4436 unusable sources, leaving 288 usable sources processCcd.calibrate.astrometry.matcher: Matched 6 sources processCcd.calibrate.astrometry.matcher WARNING: Number of matches is smaller than request processCcd.calibrate.astrometry: Matched and fit WCS in 1 iterations; found 6 matches with scatter = 0.000 +- 0.000 arcsec 

 HSC fit: visit=19696 ccd=100 2016-06-10T17:13:54: processCcd.calibrate.astrometry: Found 80 catalog sources 2016-06-10T17:13:54: processCcd.calibrate.astrometry: Matching to 119/148 good input sources 2016-06-10T17:13:55: processCcd.calibrate.astrometry: Matched 20 sources 2016-06-10T17:13:55: processCcd.calibrate.astrometry WARNING: Number of matches is smaller than request 2016-06-10T17:13:55: processCcd.calibrate.astrometry: 20 astrometric matches for 100, 0_31 2016-06-10T17:13:55: processCcd.calibrate.astrometry: Refitting WCS 2016-06-10T17:13:55: processCcd.calibrate.astrometry: Astrometric scatter: 0.038076 arcsec (with non-linear terms, 20 matches, 0 rejected) 

 LSST failed fit: visit=19696 ccd=103 processCcd.calibrate.astrometry.refObjLoader: Loaded 68 reference objects processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 68 stars processCcd.calibrate.astrometry.matcher: Purged 2206 unusable sources, leaving 225 usable sources processCcd.calibrate.astrometry.matcher: Matched 4 sources processCcd.calibrate.astrometry.matcher WARNING: Number of matches is smaller than request processCcd FATAL: Failed on dataId={'taiObs': '2015-01-21', 'pointing': 1116, 'visit': 19696, 'dateObs': '2015-01-21', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 103, 'expTime': 300.0}:   File "src/sip/CreateWcsWithSip.cc", line 142, in lsst::meas::astrom::sip::CreateWcsWithSip::CreateWcsWithSip(const std::vector<_RealType>&, const lsst::afw::image::Wcs&, int, const lsst::afw::geom::Box2I&, int) [with MatchT = lsst::afw::table::Match]  Number of matches less than requested sip order {0} lsst::pex::exceptions::LengthError: 'Number of matches less than requested sip order' 

 HSC fit: visit=19696 ccd=103 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Found 84 catalog sources 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Matching to 137/162 good input sources 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Matched 19 sources 2016-06-10T17:20:11: processCcd.calibrate.astrometry WARNING: Number of matches is smaller than request 2016-06-10T17:20:11: processCcd.calibrate.astrometry: 19 astrometric matches for 103, 1_31 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Refitting WCS 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Astrometric scatter: 0.086131 arcsec (with non-linear terms, 18 matches, 1 rejected) 

Other failed visit/ccd combos:
visit=19684 ccd=101
visit=30488 ccd=95: RuntimeError: Unable to match sources

This may simply be due to some threshold in the configs that is rejecting more stars on the LSST side, but this is not confirmed. Please investigate the cause of these failures.

## Attachments

1. HSC_102.png
732 kB
2. LSST_102.png
741 kB

## Activity

Hide
John Swinbank added a comment -

Assigning to Bob for now, but let's rebalance at the next sprint planning meeting. Or earlier, if it becomes necessary.

Show
John Swinbank added a comment - Assigning to Bob for now, but let's rebalance at the next sprint planning meeting. Or earlier, if it becomes necessary.
Hide
Lauren MacArthur added a comment -

I'm not sure if these config parameters behave in exactly the same way, but they have different default thresholds on HSC vs. LSST:

 HSC rejectThresh = pexConfig.RangeField(dtype=float, default=3.0, doc="Rejection threshold for Wcs fitting", min=0.0, inclusiveMin=False) 

https://github.com/lsst/meas_astrom/blob/master/python/lsst/meas/astrom/astrometry.py#L57

 LSST matchDistanceSigma = pexConfig.RangeField(  doc = "the maximum match distance is set to "  " mean_match_distance + matchDistanceSigma*std_dev_match_distance; " +  "ignored if not fitting a WCS",  dtype = float,  default = 2,  min = 0,  ) 

Show
Lauren MacArthur added a comment - I'm not sure if these config parameters behave in exactly the same way, but they have different default thresholds on HSC vs. LSST: https://github.com/HyperSuprime-Cam/pipe_tasks/blob/master/python/lsst/pipe/tasks/astrometry.py#L46 HSC rejectThresh = pexConfig.RangeField(dtype=float, default=3.0, doc="Rejection threshold for Wcs fitting", min=0.0, inclusiveMin=False) https://github.com/lsst/meas_astrom/blob/master/python/lsst/meas/astrom/astrometry.py#L57 LSST matchDistanceSigma = pexConfig.RangeField( doc = "the maximum match distance is set to " " mean_match_distance + matchDistanceSigma*std_dev_match_distance; " + "ignored if not fitting a WCS", dtype = float, default = 2, min = 0, )
Hide
Lauren MacArthur added a comment -

This may provide a clue. Just loading the calexps from HSC and LSST into ds9 and loading the SDSS catalog, you can see that the astrometry gets progressively worse on the LSST calexp as you move towards the vignetted region:

HSC_19696_102:

LSST_19696_102:

Show
Lauren MacArthur added a comment - This may provide a clue. Just loading the calexps from HSC and LSST into ds9 and loading the SDSS catalog, you can see that the astrometry gets progressively worse on the LSST calexp as you move towards the vignetted region: HSC_19696_102: LSST_19696_102:
Hide
Bob Armstrong added a comment -

In addition to the failures reported here, I noticed that the LSST code always matches fewer reference stars than HSC.

I think the primary problem is that LSST does not apply a distortion correction for the center of the search radius when getting objects from the reference catalog. A cut on the reference catalog is also done to be within the area of the ccd. Getting the center wrong removes potential sources. LSST also applies a S/N threshold which HSC does not.

You can somewhat get around not applying the distortion by not trimming the reference catalog to the area of the ccd. I was able to get all of the above failures to succeed by not trimming. Maybe this should be added as a config option. This does increase the size of the reference catalog, but I did not notice a significant difference in speed (although I didn't do a detailed comparison).

Show
Bob Armstrong added a comment - In addition to the failures reported here, I noticed that the LSST code always matches fewer reference stars than HSC. I think the primary problem is that LSST does not apply a distortion correction for the center of the search radius when getting objects from the reference catalog. A cut on the reference catalog is also done to be within the area of the ccd. Getting the center wrong removes potential sources. LSST also applies a S/N threshold which HSC does not. You can somewhat get around not applying the distortion by not trimming the reference catalog to the area of the ccd. I was able to get all of the above failures to succeed by not trimming. Maybe this should be added as a config option. This does increase the size of the reference catalog, but I did not notice a significant difference in speed (although I didn't do a detailed comparison).
Hide
Bob Armstrong added a comment -

This problem should be resolved with the fix identified in DM-2280. Since this ticket was to investigate and identify the problem, I am closing it.

Show
Bob Armstrong added a comment - This problem should be resolved with the fix identified in DM-2280 . Since this ticket was to investigate and identify the problem, I am closing it.

## People

• Assignee:
Bob Armstrong
Reporter:
Lauren MacArthur
Watchers:
Bob Armstrong, Dominique Boutigny, John Swinbank, Lauren MacArthur, Paul Price