Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23985

Cannot do linearity corrections in Gen 3 DECam processing

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      IsrTask has code to construct a linearizer (ip.isr.linearize.LinearizeBase) from Butler datasets. However, this code cannot be used at present in Gen 3 because:

      • IsrTask does not have a PrerequisiteInput for linearizer data
      • it is not clear where this data should come from in a new DECam repository. See discussions on #dm-middleware on March 16 and March 23. (In Gen 2, the linearizer is a virtual dataset handled by the CameraMapper, but everybody agrees that in Gen 3 it should be stored in the repository proper.)

      Please provide a way to make the linearizer data available in Gen 3 DECam repositories, both in from-scratch repositories (see DM-21862) and in repositories converted from Gen 2.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            Thanks for addressing my GitHub questions. I think unit tests to check the new/preferred style of linearity is handled correctly is the main thing still needed. I'm still interested in an answer for my Q2 above, and will feel better if you do a quick isrTask "integration test" run in gen2 and gen3 on some real data before hitting merge.

            Show
            mrawls Meredith Rawls added a comment - Thanks for addressing my GitHub questions. I think unit tests to check the new/preferred style of linearity is handled correctly is the main thing still needed. I'm still interested in an answer for my Q2 above, and will feel better if you do a quick isrTask "integration test" run in gen2 and gen3 on some real data before hitting merge.
            Hide
            czw Christopher Waters added a comment -

            As a result of me looking into the existing linearization tests in ip_isr (and being unhappy with them), I've added a new tests/test_linearize.py, and added a new ticket (DM-28816) to go back and put whatever tests are missing from this new file in, and to strip out the one-type files.

            Other cameras currently store all the information necessary for the linearity correction in the cameraGeom.  The gen3 conversion allows this to be moved to an external calibration, which can then be certified and tracked.  It also wraps up the LookupTable that DECam uses.

            Can you point to a gen2 repo that should work?  The data in the testdata_decam package looks like it's been faked entirely.

            Show
            czw Christopher Waters added a comment - As a result of me looking into the existing linearization tests in ip_isr (and being unhappy with them), I've added a new tests/test_linearize.py, and added a new ticket ( DM-28816 ) to go back and put whatever tests are missing from this new file in, and to strip out the one-type files. Other cameras currently store all the information necessary for the linearity correction in the cameraGeom.  The gen3 conversion allows this to be moved to an external calibration, which can then be certified and tracked.  It also wraps up the LookupTable that DECam uses. Can you point to a gen2 repo that should work?  The data in the testdata_decam package looks like it's been faked entirely.
            Hide
            czw Christopher Waters added a comment -

            I have run ap_verify.py --dataset CI-HiTS2015 --gen2 using both w_2021_06 and this ticket branch, and both seem to generate calexp files that are pixel-by-pixel identical (outputs in /project/czw/DM-23985/gen2repo-retest and gen2repo-ticket).  The new linearizer tests in ip_isr now correctly test all linearizer types for round-tripping through both FITS and yaml formats, so the curated calibration pathway through obs_decam_data should not cause any problems.

            I've also run a quick no-{bias, dark, flat} gen3 test that confirms (mostly to myself since I hadn't tested in more than a week) that the linearizers are ingested correctly into a gen3 repo and are used.

            Show
            czw Christopher Waters added a comment - I have run ap_verify.py --dataset CI-HiTS2015 --gen2 using both w_2021_06 and this ticket branch, and both seem to generate calexp files that are pixel-by-pixel identical (outputs in /project/czw/ DM-23985 /gen2repo-retest  and gen2repo-ticket ).  The new linearizer tests in ip_isr now correctly test all linearizer types for round-tripping through both FITS and yaml formats, so the curated calibration pathway through obs_decam_data should not cause any problems. I've also run a quick no-{bias, dark, flat} gen3 test that confirms (mostly to myself since I hadn't tested in more than a week) that the linearizers are ingested correctly into a gen3 repo and are used.
            Hide
            mrawls Meredith Rawls added a comment -

            Thank you for this and for the new test. I think this is done but for docstrings (see one lingering comment in obs_decam PR plus a couple missing test docstrings) and, if possible, reducing code duplication in the test you wrote (again see the PR).

            Show
            mrawls Meredith Rawls added a comment - Thank you for this and for the new test. I think this is done but for docstrings (see one lingering comment in obs_decam PR plus a couple missing test docstrings) and, if possible, reducing code duplication in the test you wrote (again see the PR).
            Hide
            mrawls Meredith Rawls added a comment -

            Thanks for adding the tests - assuming Jenkins is still happy, I am too  

            Show
            mrawls Meredith Rawls added a comment - Thanks for adding the tests - assuming Jenkins is still happy, I am too  

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Meredith Rawls
              Watchers:
              Christopher Waters, Colin Slater, Dino Bektesevic, Ian Sullivan, Jim Bosch, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: