Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17538

Generate stack-produced (rather than CP) calibration products for DECam

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_decam, pipe_tasks
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP S19-3, AP S20-5 (April), AP S20-6 (May), AP F20-1 (June)
    • Team:
      Alert Production

      Description

      This a timeboxed ticket, on the assumption that making calibration products is “easy”.

      When we get beyond about 2 days of work, we should file more tickets to describe the issues that are being encountered and what we're doing to resolve them

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment - - edited

            I constructed a master bias frame and a master g flat frame for DECam images of Starlink satellites taken in early March 2020. These are the notes I made for how to do this.

             
            How to process some raw DECam images 101

            • you need raw biases, raw flats, and raw images, and a setup stack
            • put a _mapper in a directory to turn it into a repo, and cd to there
            • make raw, calib, and calib_construction subdirectories
            • put all the raw images in the raw subdirectory
            • examine the raw image headers with, e.g., fitsheader or queryMetadata to learn their visit numbers
            • ingestImagesDecam.py . --mode=link raw/*
            • ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/defects
            • ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/crosstalk
            • constructBias.py . --calib calib --output calib_construction --id visit=biasVisit1^biasVisit2^biasVisitN --batch-type none
            • ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/BIAS/[date]/*.fits
            • constructFlat.py . --calib calib --output calib_construction --id visit=flatVisit1^flatVisit2^flatVisitN --config isr.doDark=False --batch-type none
            • ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/FLAT/[date]/[filter]/*.fits
            • make a ref_cats subdirectory and add symlinks to pan-starrs and gaia-dr2 refcats as usual
            • processCcd.py . --calib calib --rerun [rerun] --id visit=visit1^visit2^visitN --config isr.biasDataProductName='bias' isr.flatDataProductName='flat'
            • WARNING: to get processCcd to work the first time, I had to manually update the calibRegistry and calib directory to get the filters right; I filed DM-24514 to eventually address this and in the meantime put a workaround as one commit on this ticket
            • Note the default DataProductNames are cpBias and cpFlat for DECam, thus the configs
            • It ... seems ... to be working ?!?!
            Show
            mrawls Meredith Rawls added a comment - - edited I constructed a master bias frame and a master g flat frame for DECam images of Starlink satellites taken in early March 2020. These are the notes I made for how to do this.   How to process some raw DECam images 101 you need raw biases, raw flats, and raw images, and a setup stack put a _mapper in a directory to turn it into a repo, and cd to there make raw, calib, and calib_construction subdirectories put all the raw images in the raw subdirectory examine the raw image headers with, e.g., fitsheader or queryMetadata to learn their visit numbers ingestImagesDecam.py . --mode=link raw/* ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/defects ingestCuratedCalibs.py . --calib calib $OBS_DECAM_DATA_DIR/decam/crosstalk constructBias.py . --calib calib --output calib_construction --id visit=biasVisit1^biasVisit2^biasVisitN --batch-type none ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/BIAS/ [date] /*.fits constructFlat.py . --calib calib --output calib_construction --id visit=flatVisit1^flatVisit2^flatVisitN --config isr.doDark=False --batch-type none ingestCalibs.py . --calib calib --mode=link --validity 999 calib_construction/FLAT/ [date] / [filter] /*.fits make a ref_cats subdirectory and add symlinks to pan-starrs and gaia-dr2 refcats as usual processCcd.py . --calib calib --rerun [rerun] --id visit=visit1^visit2^visitN --config isr.biasDataProductName='bias' isr.flatDataProductName='flat' WARNING : to get processCcd to work the first time, I had to manually update the calibRegistry and calib directory to get the filters right; I filed DM-24514 to eventually address this and in the meantime put a workaround as one commit on this ticket Note the default DataProductNames are cpBias and cpFlat for DECam, thus the configs It ... seems ... to be working ?!?!
            Hide
            mrawls Meredith Rawls added a comment - - edited

            I updated the notes above as I revisited the process and worked to revise obs_decam's ingestCalibs. I can successfully build obs_decam on lsst-dev with scons and have used it to ingest both CP and stack-built calibs.

            Jenkins is in progress: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31658/pipeline 

            An example of the stack-built calibs are on lsst-dev in /project/mrawls/starlink/calib_construction and the ingested results are in /project/mrawls/starlink/calib.

            Show
            mrawls Meredith Rawls added a comment - - edited I updated the notes above as I revisited the process and worked to revise obs_decam's ingestCalibs. I can successfully build obs_decam on lsst-dev with scons and have used it to ingest both CP and stack-built calibs. Jenkins is in progress: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31658/pipeline   An example of the stack-built calibs are on lsst-dev in /project/mrawls/starlink/calib_construction and the ingested results are in /project/mrawls/starlink/calib.
            Hide
            mrawls Meredith Rawls added a comment - - edited

            I finally have the tests done for this one. There are now 3 PRs.

            I once again manually verified I can ingest both stack-built calibs and CP calibs on this ticket branch.

            I also started a new (new) Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32158/pipeline 

            Can you please look again John Parejko?

            Show
            mrawls Meredith Rawls added a comment - - edited I finally have the tests done for this one. There are now 3 PRs. I once again manually verified I can ingest both stack-built calibs and CP calibs on this ticket branch. I also started a new (new) Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32158/pipeline   Can you please look again John Parejko ?
            Hide
            mrawls Meredith Rawls added a comment -
            Show
            mrawls Meredith Rawls added a comment - Addressed all comments on GitHub and started Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32202/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the updates and new tests. Now we have something to test future gen3 code against, too.

            Show
            Parejkoj John Parejko added a comment - Thanks for the updates and new tests. Now we have something to test future gen3 code against, too.

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              swinbank John Swinbank
              Reviewers:
              John Parejko
              Watchers:
              John Parejko, John Swinbank, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.