Enable gen3 inter-chip crosstalk for DECam

XMLWordPrintable

Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
8
• Epic Link:
• Team:
Data Release Production

Description

Inter-chip crosstalk on DECam requires the ability to select with a lookup function the inputs needed for correction.  This has in turn depended on middleware updates to allow Inputs and PrerequisiteInputs to function the same way.  This ticket is a reminder so this doesn't get forgotten.

Attachments

1. DM-27833.txt
2 kB
2. pipeline_partA.png
311 kB
3. pipeline_partB.png
282 kB

Activity

Hide
Christopher Waters added a comment -

From discussions on slack, it seems that inter-chip crosstalk is still only possible in the two-step process; Step 1 processes the entire focal plane through overscan to create crosstalkSources, Step 2 then runs the standard ISR processing, but will now find the crosstalkSources.

For testing, I used the same toy repository created for DM-23985.

  pipetask qgraph -d "exposure IN (292404)" -b ./gen3 \  -i DECam/raw/all,DECam/calib \  -o czw_ct_passA -p ./IsrCt.yaml --qgraph-dot ./pipeline_partA.dot pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \  -i DECam/raw/all,DECam/calib \  -o czw_ct_passA -p ./IsrCt.yaml --register-dataset-types   pipetask qgraph -d "detector=1 AND exposure IN (292404)" -b ./gen3 \  -i czw_ct_passA \  -o czw_ct_passB -p ./IsrCtB.yaml --qgraph-dot ./pipeline_partB.dot pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \  -i czw_ct_passA \  -o czw_ct_passB -p ./IsrCtB.yaml --register-dataset-types  

I've attached the pipeline qgraph files to this ticket (they are large and wide).

This line of text here to make it clear that these are two separate pipeline qgraph files, which isn't initially obvious.

Show
Christopher Waters added a comment - From discussions on slack, it seems that inter-chip crosstalk is still only possible in the two-step process; Step 1 processes the entire focal plane through overscan to create crosstalkSources, Step 2 then runs the standard ISR processing, but will now find the crosstalkSources. For testing, I used the same toy repository created for DM-23985 . pipetask qgraph -d "exposure IN (292404)" -b ./gen3 \ -i DECam/raw/all,DECam/calib \ -o czw_ct_passA -p ./IsrCt.yaml --qgraph-dot ./pipeline_partA.dot pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \ -i DECam/raw/all,DECam/calib \ -o czw_ct_passA -p ./IsrCt.yaml --register-dataset-types   pipetask qgraph -d "detector=1 AND exposure IN (292404)" -b ./gen3 \ -i czw_ct_passA \ -o czw_ct_passB -p ./IsrCtB.yaml --qgraph-dot ./pipeline_partB.dot pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \ -i czw_ct_passA \ -o czw_ct_passB -p ./IsrCtB.yaml --register-dataset-types I've attached the pipeline qgraph files to this ticket (they are large and wide). This line of text here to make it clear that these are two separate pipeline qgraph files, which isn't initially obvious.
Hide
Meredith Rawls added a comment -

Sorry for the long delay on this review! I deeply appreciate you doing this, as it will help a lot as AP continues adopting gen3.

Two main things. First, I don't see any changes to crosstalk tests on this ticket, so please convince me they're not needed or add some. Second, I appreciate the example above for how to run this, but this workflow isn't reflected clearly in the obs_decam/pipelines directory.

One bigger meta-question, which I think I know the answer to... Are two pipetask run calls absolutely needed? I fear this will throw a wrench in ap_pipe, which runs from raws through difference imaging.

Finally, since it's been a bit (sorry!!) I'd appreciate one final "integration test" in gen2 and gen3 to make sure this works as anticipated. Happy to help with this; I have repos you are welcome to borrow at /project/mrawls/hits2015 (gen2) and /project/mrawls/hits2015-3 (gen3).

Show
Meredith Rawls added a comment - Sorry for the long delay on this review! I deeply appreciate you doing this, as it will help a lot as AP continues adopting gen3. Two main things. First, I don't see any changes to crosstalk tests on this ticket, so please convince me they're not needed or add some. Second, I appreciate the example above for how to run this, but this workflow isn't reflected clearly in the obs_decam/pipelines directory. One bigger meta-question, which I think I know the answer to... Are two pipetask run calls absolutely needed? I fear this will throw a wrench in ap_pipe, which runs from raws through difference imaging. Finally, since it's been a bit (sorry!!) I'd appreciate one final "integration test" in gen2 and gen3 to make sure this works as anticipated. Happy to help with this; I have repos you are welcome to borrow at /project/mrawls/hits2015 (gen2) and /project/mrawls/hits2015-3 (gen3).
Hide
Meredith Rawls added a comment -

The code changes look great! The pipeline situation is much clearer now, thanks.

Please make sure all tests pass when testdata_decam is setup, and as I said before, that you do a small DECam isrTask "integration test" in both gen2 and gen3. I'll be happy to mark this as reviewed when you confirm all that works! I'm willing to pair-test it with you, if you'd like.

Show
Meredith Rawls added a comment - The code changes look great! The pipeline situation is much clearer now, thanks. Please make sure all tests pass when testdata_decam is setup, and as I said before, that you do a small DECam isrTask "integration test" in both gen2 and gen3. I'll be happy to mark this as reviewed when you confirm all that works! I'm willing to pair-test it with you, if you'd like.
Hide
Christopher Waters added a comment -

I've attached the list of test commands run (DM-27833.txt), and that should cover all the integration tests I could think of.  Briefly:

• confirm obs_decam builds with testdata_decam setup
• Run }}{{ap_verify on both ticket and w10, and confirm they yield pixel-identical outputs.
• Create fresh gen2 repo, ingest the crosstalk calibs, and confirm it runs cleanly, and that the outputs have crosstalk features removed.
• Do the same in gen3, running the two step processing.
Show
Christopher Waters added a comment - I've attached the list of test commands run ( DM-27833.txt ), and that should cover all the integration tests I could think of.  Briefly: confirm obs_decam builds with  testdata_decam setup Run }}{{ap_verify on both ticket and w10 , and confirm they yield pixel-identical outputs. Create fresh gen2 repo, ingest the crosstalk calibs, and confirm it runs cleanly, and that the outputs have crosstalk features removed. Do the same in gen3, running the two step processing.
Hide
Meredith Rawls added a comment -

Outstanding! Thanks for testing all that, and for your hard work implementing this.

Show
Meredith Rawls added a comment - Outstanding! Thanks for testing all that, and for your hard work implementing this.

People

Assignee:
Christopher Waters
Reporter:
Christopher Waters
Reviewers:
Meredith Rawls
Watchers:
Christopher Waters, Ian Sullivan, Meredith Rawls, Nate Lust, Yusra AlSayyad
Votes:
0 Vote for this issue
Watchers:
5 Start watching this issue

Dates

Created:
Updated:
Resolved:

Jenkins

No builds found.