# Crosstalk sources dataset does not cleanly fit Gen3 butler concept

XMLWordPrintable

## Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
18
• Team:
Data Release Production

## Description

For cameras with inter-CCD crosstalk, the crosstalkSources dataset is produced from data on other images.  This does not cleanly fit in the Gen3 butler concept, as the required data is not necessarily known before running the butler query.

This does not prevent ci_hsc from working, as HSC has only intra-CCD CT, which does not require additional data.

## Activity

Hide
Jim Bosch added a comment - - edited

One promising idea for how to solve this (discussed recently with Christopher Waters and Nate Lust) would be to provide a way for a concrete PipelineTaskConnections implementation to customize a Prerequisite lookup.  Nominally, this would take the form of a callable that is given a Registry instance, the data ID of the quantum, and the DatasetType of the prerequisite, and returns a list of DatasetRefs; the default implementation would just call Registry.queryDatasets, as the code in pipe_base does now.  The DECam ISR customization would return all (existing) raws for that exposure, and use DeferredDatasetHandle to only load those it actually needs in runQuantum.  One catch here is that we would like it to be possible for only the DECam ISR to do this, which may imply that we want this callback to be something we can configure somehow.  A generic-ISR option for "hasMultiDetectorCrosstalk" that only DECam would enable (for now) might be easier, and would make sense.

That would also provide a workaround for the problem with jointcal QG generation (DM-21904): it would let us declare all of jointcal's inputs as prerequisites, and use the same customization hook to query for all detectors that in any exposure that overlaps the tract, rather than just the exposure-detector combinations that do.  The converse is also true: a more complete solution to the jointcal problem on DM-21904 is likely to provide a solution for DECam crosstalk as well, but I'm going to assume we'll do this ticket first since it's probably easier, even though the changes here may ultimately be obsoleted by those on DM-21904.

I'm assigning this to Nate Lust as the first part of the work will be in the Connections APIs and the QG-generation logic; I expect he'll at least collaborate with Christopher Waters on the IsrTask changes and testing with DECam.

Show
Jim Bosch added a comment - - edited One promising idea for how to solve this (discussed recently with Christopher Waters and Nate Lust ) would be to provide a way for a concrete PipelineTaskConnections implementation to customize a Prerequisite lookup.  Nominally, this would take the form of a callable that is given a Registry instance, the data ID of the quantum, and the DatasetType of the prerequisite, and returns a list of DatasetRefs; the default implementation would just call Registry.queryDatasets , as the code in pipe_base does now.  The DECam ISR customization would return all (existing) raws for that exposure, and use DeferredDatasetHandle to only load those it actually needs in runQuantum .  One catch here is that we would like it to be possible for only the DECam ISR to do this, which may imply that we want this callback to be something we can configure somehow.  A generic-ISR option for "hasMultiDetectorCrosstalk" that only DECam would enable (for now) might be easier, and would make sense. That would also provide a workaround for the problem with jointcal QG generation ( DM-21904 ): it would let us declare all of jointcal's inputs as prerequisites, and use the same customization hook to query for all detectors that in any exposure that overlaps the tract, rather than just the exposure-detector combinations that do.  The converse is also true: a more complete solution to the jointcal problem on DM-21904 is likely to provide a solution for DECam crosstalk as well, but I'm going to assume we'll do this ticket first since it's probably easier, even though the changes here may ultimately be obsoleted by those on DM-21904 . I'm assigning this to Nate Lust as the first part of the work will be in the Connections APIs and the QG-generation logic; I expect he'll at least collaborate with Christopher Waters on the IsrTask changes and testing with DECam.
Hide
Jim Bosch added a comment -

Minor comments on the PR. Only major concern (which may be a question for Christopher Waters) is whether we need another ticket to actually use this functionality in ISR as per the original ticket description. I had misremembered this as being for fringes, not crosstalk, and while it still may enable cleanups for fringes, if we actually need it to fix crosstalk we should make sure that work does eventually get done.

Show
Jim Bosch added a comment - Minor comments on the PR. Only major concern (which may be a question for Christopher Waters ) is whether we need another ticket to actually use this functionality in ISR as per the original ticket description. I had misremembered this as being for fringes, not crosstalk, and while it still may enable cleanups for fringes, if we actually need it to fix crosstalk we should make sure that work does eventually get done.
Hide
Christopher Waters added a comment -

I think we should have another ticket for using this in ISR.  I think decam CT is the only place that requires this, but being able to test that would be good (and I think this would bring gen3 ISR to feature parity with gen2).

Show
Christopher Waters added a comment - I think we should have another ticket for using this in ISR.  I think decam CT is the only place that requires this, but being able to test that would be good (and I think this would bring gen3 ISR to feature parity with gen2).
Hide

Ticket scope expanded to unblock jointcal/fgcm gen3 conversion and adding testing infrastructure to ci_hsc_gen3

Show
Yusra AlSayyad added a comment - Ticket scope expanded to unblock jointcal/fgcm gen3 conversion and adding testing infrastructure to ci_hsc_gen3

## People

• Assignee:
Nate Lust
Reporter:
Christopher Waters
Reviewers:
Jim Bosch
Watchers:
Christopher Waters, Jim Bosch, John Parejko, Krzysztof Findeisen, Yusra AlSayyad
0 Vote for this issue
Watchers:
5 Start watching this issue

## Dates

• Created:
Updated:
Resolved: