Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17169

Crosstalk sources dataset does not cleanly fit Gen3 butler concept

    Details

      Description

      For cameras with inter-CCD crosstalk, the crosstalkSources dataset is produced from data on other images.  This does not cleanly fit in the Gen3 butler concept, as the required data is not necessarily known before running the butler query.

       

      This does not prevent `ci_hsc` from working, as HSC has only intra-CCD CT, which does not require additional data.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment - - edited

            One promising idea for how to solve this (discussed recently with Christopher Waters and Nate Lust) would be to provide a way for a concrete PipelineTaskConnections implementation to customize a Prerequisite lookup.  Nominally, this would take the form of a callable that is given a Registry instance, the data ID of the quantum, and the DatasetType of the prerequisite, and returns a list of DatasetRefs; the default implementation would just call Registry.queryDatasets, as the code in pipe_base does now.  The DECam ISR customization would return all (existing) raws for that exposure, and use DeferredDatasetHandle to only load those it actually needs in runQuantum.  One catch here is that we would like it to be possible for only the DECam ISR to do this, which may imply that we want this callback to be something we can configure somehow.  A generic-ISR option for "hasMultiDetectorCrosstalk" that only DECam would enable (for now) might be easier, and would make sense.

            That would also provide a workaround for the problem with jointcal QG generation (DM-21904): it would let us declare all of jointcal's inputs as prerequisites, and use the same customization hook to query for all detectors that in any exposure that overlaps the tract, rather than just the exposure-detector combinations that do.  The converse is also true: a more complete solution to the jointcal problem on DM-21904 is likely to provide a solution for DECam crosstalk as well, but I'm going to assume we'll do this ticket first since it's probably easier, even though the changes here may ultimately be obsoleted by those on DM-21904.

            I'm assigning this to Nate Lust as the first part of the work will be in the Connections APIs and the QG-generation logic; I expect he'll at least collaborate with Christopher Waters on the IsrTask changes and testing with DECam.

            Show
            jbosch Jim Bosch added a comment - - edited One promising idea for how to solve this (discussed recently with Christopher Waters and Nate Lust ) would be to provide a way for a concrete PipelineTaskConnections implementation to customize a Prerequisite lookup.  Nominally, this would take the form of a callable that is given a Registry instance, the data ID of the quantum, and the DatasetType of the prerequisite, and returns a list of DatasetRefs; the default implementation would just call Registry.queryDatasets , as the code in pipe_base does now.  The DECam ISR customization would return all (existing) raws for that exposure, and use DeferredDatasetHandle to only load those it actually needs in runQuantum .  One catch here is that we would like it to be possible for only the DECam ISR to do this, which may imply that we want this callback to be something we can configure somehow.  A generic-ISR option for "hasMultiDetectorCrosstalk" that only DECam would enable (for now) might be easier, and would make sense. That would also provide a workaround for the problem with jointcal QG generation ( DM-21904 ): it would let us declare all of jointcal's inputs as prerequisites, and use the same customization hook to query for all detectors that in any exposure that overlaps the tract, rather than just the exposure-detector combinations that do.  The converse is also true: a more complete solution to the jointcal problem on DM-21904 is likely to provide a solution for DECam crosstalk as well, but I'm going to assume we'll do this ticket first since it's probably easier, even though the changes here may ultimately be obsoleted by those on DM-21904 . I'm assigning this to Nate Lust as the first part of the work will be in the Connections APIs and the QG-generation logic; I expect he'll at least collaborate with Christopher Waters on the IsrTask changes and testing with DECam.
            Hide
            jbosch Jim Bosch added a comment -

            Minor comments on the PR. Only major concern (which may be a question for Christopher Waters) is whether we need another ticket to actually use this functionality in ISR as per the original ticket description. I had misremembered this as being for fringes, not crosstalk, and while it still may enable cleanups for fringes, if we actually need it to fix crosstalk we should make sure that work does eventually get done.

            Show
            jbosch Jim Bosch added a comment - Minor comments on the PR. Only major concern (which may be a question for Christopher Waters ) is whether we need another ticket to actually use this functionality in ISR as per the original ticket description. I had misremembered this as being for fringes, not crosstalk, and while it still may enable cleanups for fringes, if we actually need it to fix crosstalk we should make sure that work does eventually get done.
            Hide
            czw Christopher Waters added a comment -

            I think we should have another ticket for using this in ISR.  I think decam CT is the only place that requires this, but being able to test that would be good (and I think this would bring gen3 ISR to feature parity with gen2).

            Show
            czw Christopher Waters added a comment - I think we should have another ticket for using this in ISR.  I think decam CT is the only place that requires this, but being able to test that would be good (and I think this would bring gen3 ISR to feature parity with gen2).
            Hide
            yusra Yusra AlSayyad added a comment -

            Ticket scope expanded to unblock jointcal/fgcm gen3 conversion and adding testing infrastructure to ci_hsc_gen3

            Show
            yusra Yusra AlSayyad added a comment - Ticket scope expanded to unblock jointcal/fgcm gen3 conversion and adding testing infrastructure to ci_hsc_gen3

              People

              • Assignee:
                nlust Nate Lust
                Reporter:
                czw Christopher Waters
                Reviewers:
                Jim Bosch
                Watchers:
                Christopher Waters, Jim Bosch, John Parejko, Krzysztof Findeisen, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel