Details
-
Type:
Improvement
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Story Points:18
-
Epic Link:
-
Team:Data Release Production
Description
For cameras with inter-CCD crosstalk, the crosstalkSources dataset is produced from data on other images. This does not cleanly fit in the Gen3 butler concept, as the required data is not necessarily known before running the butler query.
This does not prevent `ci_hsc` from working, as HSC has only intra-CCD CT, which does not require additional data.
Attachments
Issue Links
- blocks
-
DM-21862 Extend Gen3 butler support for obs_decam
- Done
-
DM-23616 Run converted ap_verify testdata through gen3 pipeline
- Done
- FF-depends on
-
DM-16467 isrTask conversion to pipelineTask
- Done
- relates to
-
DM-23983 Cannot apply crosstalk in Gen 3 DECam processing
- Done
-
DM-21904 Improve QuantumGraph generation for Jointcal
- In Progress
-
DM-23616 Run converted ap_verify testdata through gen3 pipeline
- Done
One promising idea for how to solve this (discussed recently with Christopher Waters and Nate Lust) would be to provide a way for a concrete PipelineTaskConnections implementation to customize a Prerequisite lookup. Nominally, this would take the form of a callable that is given a Registry instance, the data ID of the quantum, and the DatasetType of the prerequisite, and returns a list of DatasetRefs; the default implementation would just call Registry.queryDatasets, as the code in pipe_base does now. The DECam ISR customization would return all (existing) raws for that exposure, and use DeferredDatasetHandle to only load those it actually needs in runQuantum. One catch here is that we would like it to be possible for only the DECam ISR to do this, which may imply that we want this callback to be something we can configure somehow. A generic-ISR option for "hasMultiDetectorCrosstalk" that only DECam would enable (for now) might be easier, and would make sense.
That would also provide a workaround for the problem with jointcal QG generation (DM-21904): it would let us declare all of jointcal's inputs as prerequisites, and use the same customization hook to query for all detectors that in any exposure that overlaps the tract, rather than just the exposure-detector combinations that do. The converse is also true: a more complete solution to the jointcal problem on DM-21904 is likely to provide a solution for DECam crosstalk as well, but I'm going to assume we'll do this ticket first since it's probably easier, even though the changes here may ultimately be obsoleted by those on DM-21904.
I'm assigning this to Nate Lust as the first part of the work will be in the Connections APIs and the QG-generation logic; I expect he'll at least collaborate with Christopher Waters on the IsrTask changes and testing with DECam.