Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27833

Enable gen3 inter-chip crosstalk for DECam

    XMLWordPrintable

Details

    • 8
    • Data Release Production

    Description

      Inter-chip crosstalk on DECam requires the ability to select with a lookup function the inputs needed for correction.  This has in turn depended on middleware updates to allow Inputs and PrerequisiteInputs to function the same way.  This ticket is a reminder so this doesn't get forgotten.

      Attachments

        1. DM-27833.txt
          2 kB
        2. pipeline_partA.png
          pipeline_partA.png
          311 kB
        3. pipeline_partB.png
          pipeline_partB.png
          282 kB

        Issue Links

          Activity

            From discussions on slack, it seems that inter-chip crosstalk is still only possible in the two-step process; Step 1 processes the entire focal plane through overscan to create crosstalkSources, Step 2 then runs the standard ISR processing, but will now find the crosstalkSources.

            For testing, I used the same toy repository created for DM-23985.

             pipetask qgraph  -d "exposure IN (292404)" -b ./gen3 \
                      -i DECam/raw/all,DECam/calib \
                     -o czw_ct_passA -p ./IsrCt.yaml --qgraph-dot ./pipeline_partA.dot
            pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \
                     -i DECam/raw/all,DECam/calib \
                     -o czw_ct_passA -p ./IsrCt.yaml --register-dataset-types
             
            pipetask qgraph  -d "detector=1 AND exposure IN (292404)" -b ./gen3 \
                      -i czw_ct_passA \
                     -o czw_ct_passB -p ./IsrCtB.yaml --qgraph-dot ./pipeline_partB.dot
            pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \
                     -i czw_ct_passA \
                     -o czw_ct_passB -p ./IsrCtB.yaml --register-dataset-types
            
            

            I've attached the pipeline qgraph files to this ticket (they are large and wide).

            This line of text here to make it clear that these are two separate pipeline qgraph files, which isn't initially obvious.

            czw Christopher Waters added a comment - From discussions on slack, it seems that inter-chip crosstalk is still only possible in the two-step process; Step 1 processes the entire focal plane through overscan to create crosstalkSources, Step 2 then runs the standard ISR processing, but will now find the crosstalkSources. For testing, I used the same toy repository created for DM-23985 . pipetask qgraph -d "exposure IN (292404)" -b ./gen3 \ -i DECam/raw/all,DECam/calib \ -o czw_ct_passA -p ./IsrCt.yaml --qgraph-dot ./pipeline_partA.dot pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \ -i DECam/raw/all,DECam/calib \ -o czw_ct_passA -p ./IsrCt.yaml --register-dataset-types   pipetask qgraph -d "detector=1 AND exposure IN (292404)" -b ./gen3 \ -i czw_ct_passA \ -o czw_ct_passB -p ./IsrCtB.yaml --qgraph-dot ./pipeline_partB.dot pipetask run -j 4 -d "exposure IN (292404)" -b ./gen3 \ -i czw_ct_passA \ -o czw_ct_passB -p ./IsrCtB.yaml --register-dataset-types I've attached the pipeline qgraph files to this ticket (they are large and wide). This line of text here to make it clear that these are two separate pipeline qgraph files, which isn't initially obvious.

            Sorry for the long delay on this review! I deeply appreciate you doing this, as it will help a lot as AP continues adopting gen3.

            Two main things. First, I don't see any changes to crosstalk tests on this ticket, so please convince me they're not needed or add some. Second, I appreciate the example above for how to run this, but this workflow isn't reflected clearly in the obs_decam/pipelines directory.

            One bigger meta-question, which I think I know the answer to... Are two pipetask run calls absolutely needed? I fear this will throw a wrench in ap_pipe, which runs from raws through difference imaging.

            Finally, since it's been a bit (sorry!!) I'd appreciate one final "integration test" in gen2 and gen3 to make sure this works as anticipated. Happy to help with this; I have repos you are welcome to borrow at /project/mrawls/hits2015 (gen2) and /project/mrawls/hits2015-3 (gen3).

            mrawls Meredith Rawls added a comment - Sorry for the long delay on this review! I deeply appreciate you doing this, as it will help a lot as AP continues adopting gen3. Two main things. First, I don't see any changes to crosstalk tests on this ticket, so please convince me they're not needed or add some. Second, I appreciate the example above for how to run this, but this workflow isn't reflected clearly in the obs_decam/pipelines directory. One bigger meta-question, which I think I know the answer to... Are two pipetask run calls absolutely needed? I fear this will throw a wrench in ap_pipe, which runs from raws through difference imaging. Finally, since it's been a bit (sorry!!) I'd appreciate one final "integration test" in gen2 and gen3 to make sure this works as anticipated. Happy to help with this; I have repos you are welcome to borrow at /project/mrawls/hits2015 (gen2) and /project/mrawls/hits2015-3 (gen3).

            The code changes look great! The pipeline situation is much clearer now, thanks.

            Please make sure all tests pass when testdata_decam is setup, and as I said before, that you do a small DECam isrTask "integration test" in both gen2 and gen3. I'll be happy to mark this as reviewed when you confirm all that works! I'm willing to pair-test it with you, if you'd like.

            mrawls Meredith Rawls added a comment - The code changes look great! The pipeline situation is much clearer now, thanks. Please make sure all tests pass when testdata_decam is setup, and as I said before, that you do a small DECam isrTask "integration test" in both gen2 and gen3. I'll be happy to mark this as reviewed when you confirm all that works! I'm willing to pair-test it with you, if you'd like.

            I've attached the list of test commands run (DM-27833.txt), and that should cover all the integration tests I could think of.  Briefly:

            • confirm obs_decam builds with testdata_decam setup
            • Run }}{{ap_verify on both ticket and w10, and confirm they yield pixel-identical outputs.
            • Create fresh gen2 repo, ingest the crosstalk calibs, and confirm it runs cleanly, and that the outputs have crosstalk features removed.
            • Do the same in gen3, running the two step processing.
            czw Christopher Waters added a comment - I've attached the list of test commands run ( DM-27833.txt ), and that should cover all the integration tests I could think of.  Briefly: confirm obs_decam builds with  testdata_decam setup Run }}{{ap_verify on both ticket and w10 , and confirm they yield pixel-identical outputs. Create fresh gen2 repo, ingest the crosstalk calibs, and confirm it runs cleanly, and that the outputs have crosstalk features removed. Do the same in gen3, running the two step processing.

            Outstanding! Thanks for testing all that, and for your hard work implementing this.

            mrawls Meredith Rawls added a comment - Outstanding! Thanks for testing all that, and for your hard work implementing this.

            People

              czw Christopher Waters
              czw Christopher Waters
              Meredith Rawls
              Christopher Waters, Ian Sullivan, Meredith Rawls, Nate Lust, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.