Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:2
-
Epic Link:
-
Sprint:DRP S21b
-
Team:Data Release Production
-
Urgent?:No
Description
Investigate the astrometry failure encountered during the Gen3 bps DC2 processing on DM-32024. The error (reported on Slack is:
_calibrate_265317_8.5407061.err: raise pipeBase.TaskError(_calibrate_265317_8.5407061.err:lsst.pipe.base.task.TaskError: Fit failed: median scatter on sky = 13.437 arcsec > 10.000 config.maxScatterArcsec |
and this is associated with:
{instrument: 'LSSTCam-imSim', detector: 8, visit: 265317, ...} |
(which is z band and R01 S22).
Note: this was not seen in previous Gen3 runs as the single frame processing for those ran only on the visit/detector combinations that explicitly overlap one of the two tracts (and this particular dataId was not included). It is not seen in recent Gen2 runs (w_2021_40: DM-32071, w_2021_36: DM-31665, w_2021_32: DM-31351) as it overlaps with the 3828 tract that was recently dropped from the weekly processing (sigh...). However, interestingly (and perhaps somewhat frighteningly), this detector did get a successful fit in the w_2021_28 Gen2 run of DM-31067. Figure out where and what changed between then and now!
I'm all for shrinking the existing threshold for declaring failures.
I'm less certain that adding a mag cut right now is a good idea - it seems safe, but we have a feature freeze precisely because lots of things that seem safe aren't.
I'm also not certain we need
DM-32128now, unless there examples where the existing logic (with the new threshold) would not have caught a catastrophic failure but the one proposed there would have; I get that it'd be nice to use the same definitions in various places, but again, I want to aim for minimal possible changes for DP0.2 at this point. I am quite skeptical that giving the fitter additional iterations will ever allow it to recover; it's a greedy optimizer, so once it starts down into a bad local minimum, it's very hard for it to get out.