Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:AP S20-3 (February), AP S20-4 (March)
-
Team:Alert Production
-
Urgent?:No
Description
To more fully test DM-22655, we should run as much of the gen3 pipeline as we can on a converted ap_verify test dataset. As a first test, lets run on ap_verify_ci_hits, instead of the much larger non-ci dataset.
I'll provide the command to run in a comment. You'll need to get three packages on branch tickets/DM-22655 and scons'd: daf_butler, obs_base, and obs_decam. If you setup ap_verify_ci_hits2015 before sconsing obs_decam, it will run a few extra tests.
Attachments
Issue Links
- is blocked by
-
- Done
-
DM-17169 Crosstalk sources dataset does not cleanly fit Gen3 butler concept
- Done
- relates to
-
DM-23985 Cannot do linearity corrections in Gen 3 DECam processing
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
- mentioned in
-
Page Loading...
Activity
I've managed to run the pipeline through to completion. In addition to the above command lines, I had to pass the following configuration flags:
Krzysztof Findeisen
added a comment - I've managed to run the pipeline through to completion. In addition to the above command lines, I had to pass the following configuration flags:
--config isr:doCrosstalk=False : workaround for DM-23983
--config isr:doLinearize=False : workaround for DM-23985
--config calibrate:doAstrometry=False --config calibrate:doPhotoCal=False : workaround for DM-23992
Code changes made on this ticket (there's enough of them that even I'm confused):
- obs_decam (70 lines)
- refactored processCcd.py into characterizeImage.py and calibrate.py, allowing these tasks to benefit from automatic overrides in Gen 3.
- standardized the obs_decam ISR overrides, deprecating processCcdCpIsr.py in the process.
- added connections overrides to isr.py
- fixed various bugs
- ap_verify_hits2015, ap_verify_ci_hits2015, ap_pipe_testdata (96 lines)
- refactored apPipe.py into calibrate.py and imageDifference.py, allowing these tasks to be overridden when running a pipeline on the corresponding data set
- added convertRepo.py for configuring conversion of the ingested dataset to Gen 3
- pipe_tasks (188 lines)
- fixed some I/O bugs in CalibrateTask for the case where doAstrometry=False
- pipe_base (77 lines)
- fixed some bugs in the PipelineTask unit test framework discovered while testing CalibrateTask
Krzysztof Findeisen
added a comment - - edited Code changes made on this ticket (there's enough of them that even I'm confused):
obs_decam (70 lines)
refactored processCcd.py into characterizeImage.py and calibrate.py , allowing these tasks to benefit from automatic overrides in Gen 3.
standardized the obs_decam ISR overrides, deprecating processCcdCpIsr.py in the process.
added connections overrides to isr.py
fixed various bugs
ap_verify_hits2015 , ap_verify_ci_hits2015 , ap_pipe_testdata (96 lines)
refactored apPipe.py into calibrate.py and imageDifference.py , allowing these tasks to be overridden when running a pipeline on the corresponding data set
added convertRepo.py for configuring conversion of the ingested dataset to Gen 3
pipe_tasks (188 lines)
fixed some I/O bugs in CalibrateTask for the case where doAstrometry=False
pipe_base (77 lines)
fixed some bugs in the PipelineTask unit test framework discovered while testing CalibrateTask
Hi John Parejko, would you be willing to review the code changes? I think they're a bit more than we expected from "a first test"...
Krzysztof Findeisen
added a comment - Hi John Parejko , would you be willing to review the code changes? I think they're a bit more than we expected from "a first test"... See comments on the PRs.
Current steps to reproduce on lsst-dev (does not include disabling any unsupported ISR steps):
setup lsst_distribsetup -vkr /scratch/krzys001/obs_decam/ # DM-23616 ticket branch# Create small Gen 2 repositoriessetup -vkr /project/krzys001/ap_verify_ci_hits2015/ingest_dataset.py --dataset CI-HiTS2015 --output gen3demo# Convert to Gen 3cd gen3demo/convert_gen2_repo_to_gen3.py lsst.obs.decam.DarkEnergyCamera --gen2root `pwd`/ingested/ --gen3root `pwd`/gen3/ --calibs `pwd`/calibingested/ --calibFilterType abstract_filter --config config/convertRepo.py# Run ProcessCcd (first time)pipetask run --input raw/DECam,calib/DECam,refcats,skymaps --output processed --instrument lsst.obs.decam.DarkEnergyCamera --butler-config gen3/butler.yaml --pipeline ${PIPE_TASKS_DIR}/pipelines/ProcessCcd.yaml --configfile calibrate:config/calibrate.py --register-dataset-types# Run ProcessCcd (subsequently)pipetask run --output processed --instrument lsst.obs.decam.DarkEnergyCamera --butler-config gen3/butler.yaml --pipeline ${PIPE_TASKS_DIR}/pipelines/ProcessCcd.yaml --configfile calibrate:config/calibrate.py --register-dataset-types --replace-run