Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
DM-23983 implements a new CrosstalkCalib class that is the first real concrete example of the abstract base class added in DM-24537. This has resulted in a cascade of necessary changes to support this, mainly impacting obs_decam. The final result of this change is that obs_decam will need less special case code, all cameras will have a common persistable crosstalk description that is functionally the same as that in use for the Defects and Curve calibrations.
The goal of this RFC is to ensure that the major change this creates in obs_decam is clearly discussed prior to it happening. Barring any objections, I'd like to close the RFC (and merge the associated ticket) at the end of next week.
The changes to support this in ip_isr are fairly minimal, with most changes being code shuffling. The only major algorithm switch allows crosstalk to be corrected for signals on arbitrary exposures, which is necessary to support inter-chip crosstalk.
obs_lsst and obs_subaru see minimal changes from this, as the obs_lsst crosstalk isn't well measured, and obs_subaru has been using a configuration defined set of coefficients. obs_subaru should be changed in the future to use a calibration from disk, but that is out of scope for this ticket.
obs_decam is not so lucky. This ticket spiraled out of the need to support crosstalk correction in gen3 with DECam data. Some minor changes in ip_isr allow IsrTask to be split, and a new obs_decam pipeline definition duplicates how crosstalk was applied in gen2: exposures for all detectors are overscan corrected, these are used as inputs to a second instance of IsrTask that also includes the full focal plane as crosstalkSources references, these references are loaded as needed based on the information stored in the CrosstalkCalib, allowing the inter-chip crosstalk to be corrected. Both generations now use the same CrosstalkTask.run() method, with the DecamCrosstalkTask now used solely for the prepCrosstalk() method in Gen2.
The previous text document describing the DECam crosstalk is now automatically processed into the appropriate yaml descriptions by obs_decam, and these descriptions have been added to the obs_decam_data package. The FITS representations (generated by ingestCuratedCalibrations.py) have been added to the testdata_decam package, to ensure that tests that use that data have the crosstalk applied (with no change needed for the test or the expected values contained within).
Attachments
Issue Links
Activity
Andrés Alejandro Plazas Malagón mentioned that and showed me some slides. I was hoping to have that be a separate ticket, largely since it isn't currently clear what new information we'll need. We do version and declare a schema name with the new IsrCalib() base class, so extending the crosstalk to contain other newly required data without breaking the tables to be created on this ticket is reasonably easy.
Thank you for bringing this up @rhl, I had the same comment. As Christopher Waters, we have been having conversations on this (the slides he mentions are the ones by Andrew Bradshaw on crosstalk by streaks in the context of Starlink, but there's probably more data from SLAC studying non-linear crosstalk in LSSTCam).
This sounds amazing. I've only taken a cursory look at the code changes on the ticket, but it looks like you've found a way to handle inter-chip crosstalk in a way that works with both gen2 and gen3, and that the whole thing requires fewer shenanigans (a la the prepCrosstalk method). That's great! Thank you!
I have just a couple clarification questions. (1) How exactly is ip_isr split, and is this always or just sometimes? (2) What do you mean by "a new obs_decam pipeline definition" - will need to do something special to properly process decam data?
prepCrosstalk still exists, and continues to do the same job in gen2. Once we switch to gen3 everywhere, we can think about deprecating that entirely.
The split in IsrTask occurs after the overscan correction: an IsrTask instance named isrOscan performs this on raw exposures across the whole focal plane (or whatever fraction is specified for processing) writing out isrOscanCorr exposures. These isrOscanCorr files are then used as the input for a second IsrTask instance (isrFinalize), and the full focal plane of these is assigned to crosstalkSources as a list of references. isrFinalize has overscan disabled, but continues the rest of the ISR processing, loading the appropriate crosstalkSources isrOscanCorr exposures as needed during crosstalk correction. The expected exposures are defined by the keys in the interChip dictionary of the CrosstalkCalib.
The pipeline definition for ISR has been pushed to the DM-23983 ticket branch of obs_decam. That defines how to generate a gen3 version of the postISRCCD exposure, and can be prepended to other pipelines to do further processing.
Christopher Waters it doesn't look like anyone is objecting to this RFC so feel free to adopt.
I hope/suspect that this is out of scope, but the LSST Camera has non-linear crosstalk (DECam has too, but I don't think that the CP pipeline handles this — I may be wrong).
Is the future need to support this something we should think about now? The way I'd do it is to use the current
cross-talk matrix approach, but rather than multiplying the "aggressor" image by the xtalk coefficient we'd put the aggressor through a lookup table first. One might hope that this is universal for a given camera, but I suspect that it'll be more complicated than that.