Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23616

Run converted ap_verify testdata through gen3 pipeline

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify, obs_decam
    • Labels:
      None
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP S20-3 (February), AP S20-4 (March)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      To more fully test DM-22655, we should run as much of the gen3 pipeline as we can on a converted ap_verify test dataset. As a first test, lets run on ap_verify_ci_hits, instead of the much larger non-ci dataset.

      I'll provide the command to run in a comment. You'll need to get three packages on branch tickets/DM-22655 and scons'd: daf_butler, obs_base, and obs_decam. If you setup ap_verify_ci_hits2015 before sconsing obs_decam, it will run a few extra tests.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22655 Genericize gen2to3.py to be useable with any gen2 repo

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-17169 Crosstalk sources dataset does not cleanly fit Gen3 butler concept

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-23985 Cannot do linearity corrections in Gen 3 DECam processing

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-23992 Cannot load refcats in Gen 3 DECam processing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24429 Investigate change in fracDiaSourcesToSciSources in ap_verify CI

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-17169 Crosstalk sources dataset does not cleanly fit Gen3 butler concept

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-23983 Cannot apply crosstalk in Gen 3 DECam processing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21915 Support Gen 3 ingestion of ap_verify datasets

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            8 older comments
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            Current steps to reproduce on lsst-dev (does not include disabling any unsupported ISR steps):

            setup lsst_distrib
            		 
            setup -vkr /scratch/krzys001/obs_decam/  # DM-23616 ticket branch
            		 
             
            		 
            # Create small Gen 2 repositories
            		 
            setup -vkr /project/krzys001/ap_verify_ci_hits2015/
            		 
            ingest_dataset.py --dataset CI-HiTS2015 --output gen3demo
            		 
             
            		 
            # Convert to Gen 3
            		 
            cd gen3demo/
            		 
            convert_gen2_repo_to_gen3.py lsst.obs.decam.DarkEnergyCamera --gen2root `pwd`/ingested/ --gen3root `pwd`/gen3/ --calibs `pwd`/calibingested/ --calibFilterType abstract_filter --config config/convertRepo.py
            		 
             
            		 
            # Run ProcessCcd (first time)
            		 
            pipetask run --input raw/DECam,calib/DECam,refcats,skymaps --output processed --instrument lsst.obs.decam.DarkEnergyCamera --butler-config gen3/butler.yaml --pipeline ${PIPE_TASKS_DIR}/pipelines/ProcessCcd.yaml --configfile calibrate:config/calibrate.py --register-dataset-types
            		 
             
            		 
            # Run ProcessCcd (subsequently)
            		 
            pipetask run --output processed --instrument lsst.obs.decam.DarkEnergyCamera --butler-config gen3/butler.yaml --pipeline ${PIPE_TASKS_DIR}/pipelines/ProcessCcd.yaml --configfile calibrate:config/calibrate.py --register-dataset-types --replace-run

            Show
            krzys Krzysztof Findeisen added a comment - - edited Current steps to reproduce on lsst-dev (does not include disabling any unsupported ISR steps): setup lsst_distrib setup -vkr /scratch/krzys001/obs_decam/ # DM-23616 ticket branch   # Create small Gen 2 repositories setup -vkr /project/krzys001/ap_verify_ci_hits2015/ ingest_dataset.py --dataset CI-HiTS2015 --output gen3demo   # Convert to Gen 3 cd gen3demo/ convert_gen2_repo_to_gen3.py lsst.obs.decam.DarkEnergyCamera --gen2root `pwd`/ingested/ --gen3root `pwd`/gen3/ --calibs `pwd`/calibingested/ --calibFilterType abstract_filter --config config/convertRepo.py   # Run ProcessCcd (first time) pipetask run --input raw/DECam,calib/DECam,refcats,skymaps --output processed --instrument lsst.obs.decam.DarkEnergyCamera --butler-config gen3/butler.yaml --pipeline ${PIPE_TASKS_DIR}/pipelines/ProcessCcd.yaml --configfile calibrate:config/calibrate.py --register-dataset-types   # Run ProcessCcd (subsequently) pipetask run --output processed --instrument lsst.obs.decam.DarkEnergyCamera --butler-config gen3/butler.yaml --pipeline ${PIPE_TASKS_DIR}/pipelines/ProcessCcd.yaml --configfile calibrate:config/calibrate.py --register-dataset-types --replace-run
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            I've managed to run the pipeline through to completion. In addition to the above command lines, I had to pass the following configuration flags:

            • --config isr:doCrosstalk=False: workaround for DM-23983
            • --config isr:doLinearize=False: workaround for DM-23985
            • --config calibrate:doAstrometry=False --config calibrate:doPhotoCal=False: workaround for DM-23992
            Show
            krzys Krzysztof Findeisen added a comment - I've managed to run the pipeline through to completion. In addition to the above command lines, I had to pass the following configuration flags: --config isr:doCrosstalk=False : workaround for DM-23983 --config isr:doLinearize=False : workaround for DM-23985 --config calibrate:doAstrometry=False --config calibrate:doPhotoCal=False : workaround for DM-23992
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            Code changes made on this ticket (there's enough of them that even I'm confused):

            • obs_decam (70 lines)
              • refactored processCcd.py into characterizeImage.py and calibrate.py, allowing these tasks to benefit from automatic overrides in Gen 3.
              • standardized the obs_decam ISR overrides, deprecating processCcdCpIsr.py in the process.
              • added connections overrides to isr.py
              • fixed various bugs
            • ap_verify_hits2015, ap_verify_ci_hits2015, ap_pipe_testdata (96 lines)
              • refactored apPipe.py into calibrate.py and imageDifference.py, allowing these tasks to be overridden when running a pipeline on the corresponding data set
              • added convertRepo.py for configuring conversion of the ingested dataset to Gen 3
            • pipe_tasks (188 lines)
              • fixed some I/O bugs in CalibrateTask for the case where doAstrometry=False
            • pipe_base (77 lines)
              • fixed some bugs in the PipelineTask unit test framework discovered while testing CalibrateTask
            Show
            krzys Krzysztof Findeisen added a comment - - edited Code changes made on this ticket (there's enough of them that even I'm confused): obs_decam (70 lines) refactored processCcd.py into characterizeImage.py and calibrate.py , allowing these tasks to benefit from automatic overrides in Gen 3. standardized the obs_decam ISR overrides, deprecating processCcdCpIsr.py in the process. added connections overrides to isr.py fixed various bugs ap_verify_hits2015 , ap_verify_ci_hits2015 , ap_pipe_testdata (96 lines) refactored apPipe.py into calibrate.py and imageDifference.py , allowing these tasks to be overridden when running a pipeline on the corresponding data set added convertRepo.py for configuring conversion of the ingested dataset to Gen 3 pipe_tasks (188 lines) fixed some I/O bugs in CalibrateTask for the case where doAstrometry=False pipe_base (77 lines) fixed some bugs in the PipelineTask unit test framework discovered while testing CalibrateTask
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Hi John Parejko, would you be willing to review the code changes? I think they're a bit more than we expected from "a first test"...

            Show
            krzys Krzysztof Findeisen added a comment - Hi John Parejko , would you be willing to review the code changes? I think they're a bit more than we expected from "a first test"...
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            See comments on the PRs.

            Show
            Parejkoj John Parejko added a comment - See comments on the PRs.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              John Parejko
              Watchers:
              Christopher Waters, Eric Bellm, Ian Sullivan, John Parejko, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.