Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23983

Cannot apply crosstalk in Gen 3 DECam processing

    Details

    • Story Points:
      20
    • Sprint:
      DRP S20-5 (Apr)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      IsrTask delegates complex crosstalk corrections to CrosstalkTask. However, the Gen 3 call to CrosstalkTask is disabled, citing DM-17169. Because CrosstalkTask does not run, Gen 3 calls to IsrTask will fail unless the configuration includes doCrosstalk=False.

      Please make it possible to run CrosstalkTask in Gen 3 and re-enable it.

        Attachments

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment -

            Consultation with Nate Lust has indicated that my strategy for the two-stage ISR will work fine after DM-21904.  However, this means that until that ticket is finished, these two stages will need to be run in separate pipelines.  I've added a ticket to resolve this, and after rebasing and testing the current code with the split pipelines, will merge this ticket.

            Show
            czw Christopher Waters added a comment - Consultation with Nate Lust has indicated that my strategy for the two-stage ISR will work fine after DM-21904 .  However, this means that until that ticket is finished, these two stages will need to be run in separate pipelines.  I've added a ticket to resolve this, and after rebasing and testing the current code with the split pipelines, will merge this ticket.
            Hide
            czw Christopher Waters added a comment -

            Description of the middleware issue:

            IsrTaskConnections.ccdExposure is now an pipeBase.connectionType.Input.  Passing the same dataset type to this and IsrTaskConnections.crosstalkSources causes the quantum graph generation to fail, because that is a PrerequisiteInput, and the same dataset is not allowed to have different connectionTypes.  Switching crosstalkSources to an Input causes a different problem: we need to select all existing inputs that have the same dataId as the ccdExposure, but without the "detector" constraint (to load the full focal plane of possible crosstalk source exposures).  This was being done with a lookupFunction that could remove the "detector" constraint before querying the butler.  This method only exists for PrerequisiteInput.

            To resolve this ticket, I am keeping crosstalkSources as a PrerequisiteInput, and removing the two-step pipeline definition for obs_decam.  This breaks inter-chip crosstalk in gen3, which will have to wait for improvements in quantum graph generation that are already planned.  DM-25348 will contain the inter-chip fix, and will be done once the middleware tickets merge.

            Show
            czw Christopher Waters added a comment - Description of the middleware issue: IsrTaskConnections.ccdExposure is now an pipeBase.connectionType.Input.  Passing the same dataset type to this and IsrTaskConnections.crosstalkSources causes the quantum graph generation to fail, because that is a PrerequisiteInput, and the same dataset is not allowed to have different connectionTypes.  Switching crosstalkSources to an Input causes a different problem: we need to select all existing inputs that have the same dataId as the ccdExposure, but without the "detector" constraint (to load the full focal plane of possible crosstalk source exposures).  This was being done with a lookupFunction that could remove the "detector" constraint before querying the butler.  This method only exists for PrerequisiteInput. To resolve this ticket, I am keeping crosstalkSources as a PrerequisiteInput, and removing the two-step pipeline definition for obs_decam.  This breaks inter-chip crosstalk in gen3, which will have to wait for improvements in quantum graph generation that are already planned.  DM-25348 will contain the inter-chip fix, and will be done once the middleware tickets merge.
            Hide
            mrawls Meredith Rawls added a comment -

            Understood, thanks for the update. I'm a little sad that gen3 inter-CCD crosstalk won't happen after all on this ticket, but you've done a lot of important work here to get gen2 to work as before as well as implementing gen3 intra-CCD crosstalk. Happy to take a final look if there have been significant code changes since the initial review, just let me know.

            This isn't the only situation where we will need a separate pipeline to run to generate "prerequisite" data products, so I'm glad that's being handled in DM-21904! I'd be happy to review the new followup DM-25348 when it's ready, and we'll make do without gen3 inter-CCD crosstalk for now.

            Show
            mrawls Meredith Rawls added a comment - Understood, thanks for the update. I'm a little sad that gen3 inter-CCD crosstalk won't happen after all on this ticket, but you've done a lot of important work here to get gen2 to work as before as well as implementing gen3 intra-CCD crosstalk. Happy to take a final look if there have been significant code changes since the initial review, just let me know. This isn't the only situation where we will need a separate pipeline to run to generate "prerequisite" data products, so I'm glad that's being handled in DM-21904 ! I'd be happy to review the new followup DM-25348 when it's ready, and we'll make do without gen3 inter-CCD crosstalk for now.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Christopher Waters so what does this mean for running DECam in Gen 3? Will we still have crashes as originally reported, or will we simply have lower-quality outputs because not all of the crosstalk has been corrected?

            Show
            krzys Krzysztof Findeisen added a comment - Christopher Waters so what does this mean for running DECam in Gen 3? Will we still have crashes as originally reported, or will we simply have lower-quality outputs because not all of the crosstalk has been corrected?
            Hide
            czw Christopher Waters added a comment -

            There should be no crashes from crosstalk with DECam in Gen3 (at least, I am unable to find inputs that crash on my ticket branch runs).  The outputs will be lower quality as they are not corrected for inter-chip crosstalk, but will have the intra-chip (amp-to-amp in the same detector) corrected.

             

            Show
            czw Christopher Waters added a comment - There should be no crashes from crosstalk with DECam in Gen3 (at least, I am unable to find inputs that crash on my ticket branch runs).  The outputs will be lower quality as they are not corrected for inter-chip crosstalk, but will have the intra-chip (amp-to-amp in the same detector) corrected.  

              People

              • Assignee:
                czw Christopher Waters
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Meredith Rawls
                Watchers:
                Christopher Waters, Dino Bektesevic, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: