Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34863

Convert tests in obs_decam from gen2 to gen3 and remove gen2.

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_decam
    • Labels:

      Description

      After DM-34862, the tests in obs_decam can be migrated from gen2 to gen3.

      These tests should use the SimplePipelineExecutor to do calibration lookups and exercise basic ISR functionality including crosstalk. The current nopytest_test_processCcd.py is slow (it goes all the way to calexp) and mostly unnecessary and will be removed. Within that test file is testWcsPostIsr which is essential to maintain. This ticket will create a new test_isr.py which will replace test_crosstalk.py and include the necessary crosstalk and wcs (and any other tests that require decam ISR).

      Other tests that touch gen2-only features (like testing mappers) will be removed, and tests will be migrated to gen3 as appropriate.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35035 Remove gen2 support from obs_base

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33254 Remove Gen 2 support from ap_pipe

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34862 Migrate testdata_decam to gen3

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35655 Remove gen2 jointcal code and tests

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33523 Remove gen2 support from obs_cfht

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            You can also remove the gen2 code on this ticket if you like.

            Show
            tjenness Tim Jenness added a comment - You can also remove the gen2 code on this ticket if you like.
            Hide
            Permalink
            lskelvin Lee Kelvin added a comment -

            Thanks for going through this Eli, this ticket is in great shape. Old gen2 docs/code have been removed and unit tests have been updated to gen3. I don't have any major comments to add on the PR.

            As a test, I git cloned and built obs_decam on the main branch and on this ticket branch. The main branch builds in ~6m, with an install size of 382M. This ticket branch builds in ~3m with an install size of 184M, so a significant improvement on both fronts.

            As a further test, I ran my example Merian DECam data through step 1 processing. The run command is similar to this:

            pipetask --long-log run --register-dataset-types -j 12
            		 
            -b /project/lskelvin/repo --instrument lsst.obs.decam.DarkEnergyCamera
            		 
            -i DECam/defaults/merian9813
            		 
            -o u/lskelvin/scratch/tickets/DM-34863-ticket
            		 
            -p $DRP_PIPE_DIR/pipelines/DECam/DRP-Merian.yaml#step1
            		 
            -d “instrument=‘DECam’ AND visit=971666”

            I confirm that the PVI outputs are identical between the vanilla and ticket branch outputs, so no cause for concern there.

            Assuming Jenkins doesn't complain, this look good to merge to me. Thanks again Eli!

            Show
            lskelvin Lee Kelvin added a comment - Thanks for going through this Eli, this ticket is in great shape. Old gen2 docs/code have been removed and unit tests have been updated to gen3. I don't have any major comments to add on the PR. As a test, I git cloned and built obs_decam on the main branch and on this ticket branch. The main branch builds in ~6m, with an install size of 382M. This ticket branch builds in ~3m with an install size of 184M, so a significant improvement on both fronts. As a further test, I ran my example Merian DECam data through step 1 processing. The run command is similar to this: pipetask --long-log run --register-dataset-types -j 12 -b /project/lskelvin/repo --instrument lsst.obs.decam.DarkEnergyCamera -i DECam/defaults/merian9813 -o u/lskelvin/scratch/tickets/DM-34863-ticket -p $DRP_PIPE_DIR/pipelines/DECam/DRP-Merian.yaml#step1 -d “instrument=‘DECam’ AND visit=971666” I confirm that the PVI outputs are identical between the vanilla and ticket branch outputs, so no cause for concern there. Assuming Jenkins doesn't complain, this look good to merge to me. Thanks again Eli!
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Thanks so much! I think those sizes include the source + the .git tree. Looking at jenkins (from e.g. https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36966/pipeline/46) the install size goes from 300 Mb (before) to 5.6 Mb (after).

            Show
            erykoff Eli Rykoff added a comment - Thanks so much! I think those sizes include the source + the .git tree. Looking at jenkins (from e.g. https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36966/pipeline/46 ) the install size goes from 300 Mb (before) to 5.6 Mb (after).

              People

              Assignee:
              erykoff Eli Rykoff
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Lee Kelvin
              Watchers:
              Eli Rykoff, John Parejko, Lee Kelvin, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.