Details
-
Type:
Bug
-
Status: To Do
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: meas_astrom
-
Labels:None
-
Team:Data Release Production
-
Urgent?:No
Description
There are occasional silent failures of the astrometry task in processCcd, that Dominique Boutigny has long complained about on Slack: e.g., https://lsstc.slack.com/archives/C978LTJGN/p1526484056000588 but that never got ticketed.
The gist of the problem is that sometimes the astrometry fails badly (in particular in many DESC DC2 u-band ccds, but there are probably other examples) but the fitter thinks everything is fine. This has led to Dominique Boutigny's script to check for these failures: https://github.com/LSSTDESC/ImageProcessingPipelines/blob/master/python/util/checkCcdAstrometry.py (see DM-16301 for a ticket to add the logic from this script to processCcd).
Recent investigation (see, e.g., https://lsstc.slack.com/archives/C978LTJGN/p1543335472430300 and following) makes it seem like the culprit for most (all?) of these silent failures is that the fitter is trying to fit more SIP terms than there are constraints from the stars. When these stars are in only one section of the CCD, this leads to overfitting for the matched stars and wild extrapolations for other stars, leading to internal statistics from the fitter looking "good" but the overall fit being bad.
(In general, if the default order of the SIP fit is reduced to 2 this seems to greatly increase the success rate on simulated u-band images).
Specifically, for this ticket, I propose having the fitter raise an exception of there are not enough matched stars for the SIP order that was requested. This would turn silent failures into loud failures, as well as alerting the user of where things were going wrong.
Attachments
Issue Links
Activity
On closer inspection, it appears that CreateWcsWithSip.cc (line 130) already implements SIP order checking against matched star numbers (post-rejection), and so the rationale for this ticket may have been removed. Specifically, it checks that the number of matched stars is greater than order+1, which we believe to be the correct logic in this case.
To wrap up this ticket, I propose that we leave in the python-side info logger code we've added which provides the number of rejected stars if greater than zero, as this information wasn't immediately obvious beforehand.
My guess is that that cut is the bare minimum for the code not to crash, but in practice is insufficient to ensure the fit actually makes sense.
I agree - the trouble is that this ticket overlaps with DM-16301, which seemingly better addresses that issue. For the purposes of this ticket (having the fitter raise an exception if there are not enough matched stars for the SIP order that was requested), it looks like this is already being done within CreateWcsWithSip.cc, so we're not sure what else can/should be added here.
Following the discussion on DM-16301, I'm also pausing this ticket until further 'successfully failing' data IDs are available for testing.
On a related note: Dominique Boutigny if you have examples of problematic fits that we can reproduce on lsst-dev, that would be very helpful.