Design/change sketch for AstrometryTask updates
I think that the below encompasses all of the necessary work to implement the new single frame astrometry design. John Swinbank: is this sufficient to close out this ticket? I'm filing a new ticket to start the work on the first part.
Reading raw Exposure changes
- This work is going to go into the Gen2 butler world. Converting it to gen3 butler will be a separate job, but could be made easier by keeping all of the relevant changes in free functions that we can just use directly.
- The initial SkyWcs is assigned in `exposureFromImage` (called from `_standardizeExposure`, called from `std_raw`, so this all happens when the raw is loaded by the butler via `std_raw`).
- `exposureFromImage` first calls `makeSkyWcs(metadata)`, and only later builds the `VisitInfo`. We will have to reorder that so that `VisitInfo` gets built first, then `makeskyWcs(pixelsToFieldAngle, orientation, flipX, boresight)` is called later, using the camerageGeom to specify `pixelsToFieldAngle` and `flipX`, and the `VisitInfo` to specify `orientation` and `boresight`.
- Need to write a separate function to call from `exposureFromImage` (`createInitialSkyWcs(visitInfo, detector)`?) that looks similar to `lsstCamMapper.getWcsFromDetector`.
- We may have a problem due to when metadata gets stripped. Jim mentioned this gen3 commit as an example. However, if we don't actually need the FITS WCS metadata at all (which we won't with the new above function), this should not matter. https://github.com/lsst/daf_butler/commit/ac5032c6361cfb3a6f8d1dd78cf2b52fb6baa3e3
- Gen3 has `FitsExposureFormatter`, which might be able to call that new function, if we put it in a good location.
- A later addition would be to add an atmospheric refraction correction to the above model, so that the matcher does not have to account for it.
- We can directly compute this, since we know the position and sky conditions of the exposure.
- This work should be a separate ticket, once the above code is stable.
How to test this?
We can test by looking at the initial matcher output for all sensors on a single visit. The matcher reports how far it had to move at each iteration; if we scatter plot that for all sensors in one visit, they should cluster pretty tightly if we've gotten the input CameraGeom correct. If the distortion model is not good, there will be much larger scatter and/or more outliers.
Jim suggests writing regression tests using `testdata_*` for the various obs packages, testing `pixelToSky` and `skyToPixel` for e.g. the ccd bounding boxes.
ISR has an optional `addDistortionModel` which appears to only be turned on for HSC. This uses afw's `makeDistortedTanWcs()`. Although Russell also wrote this, I think we'll be much better off just using `makeSkyWcs` as above: it should be much simpler than what goes on in `makeDistortedTanWcs`.
- Disable `doAddDistortionModel` for all cameras, so that our above work does not get overwritten during ISR.
- Deprecate and remove the relevant code from ISR task: there is no reason to do this in ISR instead of doing it as part of the reading of the raw exposure data.
Per discussion on slack we will start by not writing a new fitter, but just testing how close the above initial WCS fixes get us to what we want. From there, we can decide how far we want to proceed with a new fitter.
- New fitter: `FitWithFixedCameraGeomTask`?
- This new fitter might not have to iterate at all. If it is just fitting crval and cdMatrix, that might be a single step: match to refcat, compute crval (still at boresight), compute cdMatrix, done.
- We want to keep crval at the boresight so that crpix is still defined at the boresight, so that the cameraGeom is unchanged. We will have to correct crval to account for any offset relative to the matched sources.
- The fitter can use a pure linear regression, fitting cdMatrix + crval offset.
- We then set `AstrometryConfig.wcsFitter` to this and everything else should be the same?
- While we're doing this, we might want to tidy up the the `except Exception as e` line in astrometry.py to not catch everything?
- If we discover problems with getting enough refcat matches (e.g. due to too large of boresight offsets), we should just increase the padding for `loadPixelBox`.
How to test this?
Lauren is identifying test data for this process, which will be used by e.g. `ci_hsc` or `test_processCcd`. We can write unit tests of the fitting code itself with some existing testdata sets, and/or trivial fake points+fake refcats, just to check that the fitter is behaving correctly.