Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17538

Generate stack-produced (rather than CP) calibration products for DECam

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_decam, pipe_tasks
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP S19-3, AP S20-5 (April), AP S20-6 (May), AP F20-1 (June)
    • Team:
      Alert Production

      Description

      This a timeboxed ticket, on the assumption that making calibration products is “easy”.

      When we get beyond about 2 days of work, we should file more tickets to describe the issues that are being encountered and what we're doing to resolve them

        Attachments

          Issue Links

            Activity

            swinbank John Swinbank created issue -
            swinbank John Swinbank made changes -
            Field Original Value New Value
            Epic Link DM-16408 [ 233202 ]
            swinbank John Swinbank made changes -
            Rank Ranked higher
            swinbank John Swinbank made changes -
            Sprint AP S19-3 [ 831 ] AP S19-3, AP S19-4 [ 831, 832 ]
            swinbank John Swinbank made changes -
            Sprint AP S19-3, AP S19-4 [ 831, 832 ] AP S19-3, AP S19-6 [ 831, 834 ]
            swinbank John Swinbank made changes -
            Rank Ranked lower
            swinbank John Swinbank made changes -
            Epic Link DM-16408 [ 233202 ] DM-17992 [ 240548 ]
            swinbank John Swinbank made changes -
            Rank Ranked lower
            swinbank John Swinbank made changes -
            Sprint AP S19-3, AP S19-6 [ 831, 834 ] AP S19-3 [ 831 ]
            swinbank John Swinbank made changes -
            Rank Ranked higher
            swinbank John Swinbank made changes -
            Epic Link DM-17992 [ 240548 ] DM-19977 [ 307523 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-21016 [ DM-21016 ]
            swinbank John Swinbank made changes -
            Epic Link DM-19977 [ 307523 ] DM-21443 [ 423050 ]
            swinbank John Swinbank made changes -
            Epic Link DM-21443 [ 423050 ] DM-22634 [ 427743 ]
            swinbank John Swinbank made changes -
            Sprint AP S19-3 [ 831 ] AP S19-3, AP S20-5 (April) [ 831, 986 ]
            swinbank John Swinbank made changes -
            Rank Ranked lower
            swinbank John Swinbank made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            swinbank John Swinbank made changes -
            Link This issue blocks DM-24253 [ DM-24253 ]
            Hide
            mrawls Meredith Rawls added a comment - - edited

            I constructed a master bias frame and a master g flat frame for DECam images of Starlink satellites taken in early March 2020. These are the notes I made for how to do this.

             
            How to process some raw DECam images 101

            • you need raw biases, raw flats, and raw images, and a setup stack
            • put a _mapper in a directory to turn it into a repo, and cd to there
            • make raw, calib, and calib_construction subdirectories
            • put all the raw images in the raw subdirectory
            • examine the raw image headers with, e.g., fitsheader or queryMetadata to learn their visit numbers
            • ingestImagesDecam.py . --mode=link raw/*
            • ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/defects
            • ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/crosstalk
            • constructBias.py . --calib calib --output calib_construction --id visit=biasVisit1^biasVisit2^biasVisitN --batch-type none
            • ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/BIAS/[date]/*.fits
            • constructFlat.py . --calib calib --output calib_construction --id visit=flatVisit1^flatVisit2^flatVisitN --config isr.doDark=False --batch-type none
            • ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/FLAT/[date]/[filter]/*.fits
            • make a ref_cats subdirectory and add symlinks to pan-starrs and gaia-dr2 refcats as usual
            • processCcd.py . --calib calib --rerun [rerun] --id visit=visit1^visit2^visitN --config isr.biasDataProductName='bias' isr.flatDataProductName='flat'
            • WARNING: to get processCcd to work the first time, I had to manually update the calibRegistry and calib directory to get the filters right; I filed DM-24514 to eventually address this and in the meantime put a workaround as one commit on this ticket
            • Note the default DataProductNames are cpBias and cpFlat for DECam, thus the configs
            • It ... seems ... to be working ?!?!
            Show
            mrawls Meredith Rawls added a comment - - edited I constructed a master bias frame and a master g flat frame for DECam images of Starlink satellites taken in early March 2020. These are the notes I made for how to do this.   How to process some raw DECam images 101 you need raw biases, raw flats, and raw images, and a setup stack put a _mapper in a directory to turn it into a repo, and cd to there make raw, calib, and calib_construction subdirectories put all the raw images in the raw subdirectory examine the raw image headers with, e.g., fitsheader or queryMetadata to learn their visit numbers ingestImagesDecam.py . --mode=link raw/* ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/defects ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/crosstalk constructBias.py . --calib calib --output calib_construction --id visit=biasVisit1^biasVisit2^biasVisitN --batch-type none ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/BIAS/ [date] /*.fits constructFlat.py . --calib calib --output calib_construction --id visit=flatVisit1^flatVisit2^flatVisitN --config isr.doDark=False --batch-type none ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/FLAT/ [date] / [filter] /*.fits make a ref_cats subdirectory and add symlinks to pan-starrs and gaia-dr2 refcats as usual processCcd.py . --calib calib --rerun [rerun] --id visit=visit1^visit2^visitN --config isr.biasDataProductName='bias' isr.flatDataProductName='flat' WARNING : to get processCcd to work the first time, I had to manually update the calibRegistry and calib directory to get the filters right; I filed DM-24514 to eventually address this and in the meantime put a workaround as one commit on this ticket Note the default DataProductNames are cpBias and cpFlat for DECam, thus the configs It ... seems ... to be working ?!?!
            mrawls Meredith Rawls made changes -
            Link This issue relates to DM-24294 [ DM-24294 ]
            swinbank John Swinbank made changes -
            Epic Link DM-22634 [ 427743 ] DM-24344 [ 433031 ]
            Hide
            mrawls Meredith Rawls added a comment - - edited

            I updated the notes above as I revisited the process and worked to revise obs_decam's ingestCalibs. I can successfully build obs_decam on lsst-dev with scons and have used it to ingest both CP and stack-built calibs.

            Jenkins is in progress: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31658/pipeline 

            An example of the stack-built calibs are on lsst-dev in /project/mrawls/starlink/calib_construction and the ingested results are in /project/mrawls/starlink/calib.

            Show
            mrawls Meredith Rawls added a comment - - edited I updated the notes above as I revisited the process and worked to revise obs_decam's ingestCalibs. I can successfully build obs_decam on lsst-dev with scons and have used it to ingest both CP and stack-built calibs. Jenkins is in progress: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31658/pipeline   An example of the stack-built calibs are on lsst-dev in /project/mrawls/starlink/calib_construction and the ingested results are in /project/mrawls/starlink/calib.
            mrawls Meredith Rawls made changes -
            Reviewers John Parejko [ parejkoj ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            mrawls Meredith Rawls made changes -
            Story Points 4 6
            swinbank John Swinbank made changes -
            Sprint AP S19-3, AP S20-5 (April) [ 831, 986 ] AP S19-3, AP S20-5 (April), AP S20-6 (May) [ 831, 986, 987 ]
            swinbank John Swinbank made changes -
            Epic Link DM-24344 [ 433031 ] DM-25146 [ 435264 ]
            swinbank John Swinbank made changes -
            Sprint AP S19-3, AP S20-5 (April), AP S20-6 (May) [ 831, 986, 987 ] AP S19-3, AP S20-5 (April), AP S20-6 (May), AP F20-1 (June) [ 831, 986, 987, 1019 ]
            Hide
            mrawls Meredith Rawls added a comment - - edited

            I finally have the tests done for this one. There are now 3 PRs.

            I once again manually verified I can ingest both stack-built calibs and CP calibs on this ticket branch.

            I also started a new (new) Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32158/pipeline 

            Can you please look again John Parejko?

            Show
            mrawls Meredith Rawls added a comment - - edited I finally have the tests done for this one. There are now 3 PRs. I once again manually verified I can ingest both stack-built calibs and CP calibs on this ticket branch. I also started a new (new) Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32158/pipeline   Can you please look again John Parejko ?
            sullivan Ian Sullivan made changes -
            Component/s obs_decam [ 12851 ]
            Component/s pipe_tasks [ 10726 ]
            Hide
            mrawls Meredith Rawls added a comment -
            Show
            mrawls Meredith Rawls added a comment - Addressed all comments on GitHub and started Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32202/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the updates and new tests. Now we have something to test future gen3 code against, too.

            Show
            Parejkoj John Parejko added a comment - Thanks for the updates and new tests. Now we have something to test future gen3 code against, too.
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            mrawls Meredith Rawls made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                swinbank John Swinbank
                Reviewers:
                John Parejko
                Watchers:
                John Parejko, John Swinbank, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel