Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27922

Create initial gen3 calibs for DC2

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      10
    • Epic Link:
    • Team:
      Data Release Production

      Description

      I have a copy of the raw calibration frames from James Chiang in /project/czw/dataDirs/DC2_raw_calibs/, and will be ingesting these and testing gen3 calibration production.  It looks like there are bias, dark, and flat frames, which should be sufficient to produce bias, dark, flats, PTC curves, and linearity models, with brighter-fatter kernels possible when DM-24704 closes.

        Attachments

          Issue Links

          FF-depends on

          Improvement - An improvement or enhancement to an existing feature or task. DM-24704 Make brighter-fatter correction a subclass of lsst.ip.isr.IsrCalib

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-29581 runISR.py can't find defect files

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            czw Christopher Waters added a comment -

            The bias, dark, and flat worked cleanly with no additional cp_pipe code development.  The outputs can be found in /project/czw/dp0_calibs/repo/ in the calib/v00 calibration collection.

            Show
            czw Christopher Waters added a comment - The bias, dark, and flat worked cleanly with no additional cp_pipe code development.  The outputs can be found in /project/czw/dp0_calibs/repo/  in the calib/v00  calibration collection.
            Hide
            Permalink
            czw Christopher Waters added a comment - - edited

            I'll likely rerun these commands from scratch once DM-24704 merges (I'm in the process of cleaning up the code before moving it into review), but I wanted to document the commands that I used.

            DATADIR=/project/czw/dataDirs/DC2_raw_calibs/
            		 
            REPO=/project/czw/dp0_calibs/repo/
            		 
             
            		 
            butler create ${REPO}
            		 
            butler register-instrument ${REPO} lsst.obs.lsst.LsstCamImSim
            		 
            butler write-curated-calibrations ${REPO} lsst.obs.lsst.LsstCamImSim
            		 
            butler ingest-raws --transfer=symlink ${REPO} ${DATADIR}/bf_flats_20190408/
            		 
            butler ingest-raws --transfer=symlink ${REPO} ${DATADIR}/calibration_data/
            		 
             
            		 
            pipetask run -d "exposure IN (3000000,3000001,3000010,3000011,3000020, 3000021,3000022,3000023,3000024,3000025, 3000026,3000027,3000028,3000029,3000030, 3000031,3000032,3000033,3000034,3000035)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib -o u/czw/bias00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpBias.yaml --register-dataset-types
            		 
            butler certify-calibrations ${REPO} u/czw/bias00 LSSTCam-imSim/calib/dp0 --begin-date 1980-01-01 --end-date 2050-01-01 bias
            		 
             
            		 
            pipetask run -d "exposure IN (4000000,4000001,4000002,4000003,4000010, 4000011,4000012,4000013,4000014,4000015, 4000016,4000017,4000018,4000019,4000020, 4000021,4000022,4000023,4000024,4000025)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/dark00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpDark.yaml --register-dataset-types
            		 
            butler certify-calibrations ${REPO} u/czw/dark00 LSSTCam-imSim/calib/dp0 --begin-date 1980-01-01 --end-date 2050-01-01 dark
            		 
             
            		 
            pipetask --long-log run -d "exposure IN (5000110,5000120,5000130,5000140,5000150, 5000160,5000170,5000180,5000210,5000220, 5000230,5000240,5000250,5000260,5000270,                                                                 5000280,5000310,5000320,5000330,5000340, 5000350,5000360,5000370,5000380,5000410,                                                                 5000420,5000430,5000440,5000450,5000460, 5000470,5000480,5000510,5000525,5000530, 5000540,5000550,5000560,5000570,5000580)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/flat00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpFlat.yaml --register-dataset-types
            		 
            butler certify-calibrations ${REPO} u/czw/flat00 LSSTCam-imSim/calib/dp0 --begin-date 1980-01-01 --end-date 2050-01-01 flat
            		 
             
            		 
            pipetask --long-log run -d "exposure IN (5000110,5000120,5000130,5000140,5000150, 5000160,5000170,5000180,5000210,5000220, 5000230,5000240,5000250,5000260,5000270,                                                                5000280,5000310,5000320,5000330,5000340, 5000350,5000360,5000370,5000380,5000410,                                                                5000420,5000430,5000440,5000450,5000460, 5000470,5000480,5000510,5000525,5000530, 5000540,5000550,5000560,5000570,5000580)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/ptc00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpPtc.yaml --register-dataset-types
            		 
            butler certify-calibrations ${REPO} u/czw/ptc00 LSSTCam-imSim/calib/dp0 --begin-date 1980-01-01 --end-date 2050-01-01 ptc
            		 
             
            		 
            pipetask --long-log run -d " exposure IN (5000110)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/bfk00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpBfkSolve.yaml --register-dataset-types

             

            Show
            czw Christopher Waters added a comment - - edited I'll likely rerun these commands from scratch once DM-24704 merges (I'm in the process of cleaning up the code before moving it into review), but I wanted to document the commands that I used. DATADIR=/project/czw/dataDirs/DC2_raw_calibs/ REPO=/project/czw/dp0_calibs/repo/   butler create ${REPO} butler register-instrument ${REPO} lsst.obs.lsst.LsstCamImSim butler write-curated-calibrations ${REPO} lsst.obs.lsst.LsstCamImSim butler ingest-raws --transfer=symlink ${REPO} ${DATADIR}/bf_flats_20190408/ butler ingest-raws --transfer=symlink ${REPO} ${DATADIR}/calibration_data/   pipetask run -d "exposure IN (3000000,3000001,3000010,3000011,3000020, 3000021,3000022,3000023,3000024,3000025, 3000026,3000027,3000028,3000029,3000030, 3000031,3000032,3000033,3000034,3000035)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib -o u/czw/bias00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpBias.yaml --register-dataset-types butler certify-calibrations ${REPO} u/czw/bias00 LSSTCam-imSim/calib/dp0 --begin-date 1980 - 01 - 01 --end-date 2050 - 01 - 01 bias   pipetask run -d "exposure IN (4000000,4000001,4000002,4000003,4000010, 4000011,4000012,4000013,4000014,4000015, 4000016,4000017,4000018,4000019,4000020, 4000021,4000022,4000023,4000024,4000025)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/dark00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpDark.yaml --register-dataset-types butler certify-calibrations ${REPO} u/czw/dark00 LSSTCam-imSim/calib/dp0 --begin-date 1980 - 01 - 01 --end-date 2050 - 01 - 01 dark   pipetask -- long -log run -d "exposure IN (5000110,5000120,5000130,5000140,5000150, 5000160,5000170,5000180,5000210,5000220, 5000230,5000240,5000250,5000260,5000270, 5000280,5000310,5000320,5000330,5000340, 5000350,5000360,5000370,5000380,5000410, 5000420,5000430,5000440,5000450,5000460, 5000470,5000480,5000510,5000525,5000530, 5000540,5000550,5000560,5000570,5000580)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/flat00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpFlat.yaml --register-dataset-types butler certify-calibrations ${REPO} u/czw/flat00 LSSTCam-imSim/calib/dp0 --begin-date 1980 - 01 - 01 --end-date 2050 - 01 - 01 flat   pipetask -- long -log run -d "exposure IN (5000110,5000120,5000130,5000140,5000150, 5000160,5000170,5000180,5000210,5000220, 5000230,5000240,5000250,5000260,5000270, 5000280,5000310,5000320,5000330,5000340, 5000350,5000360,5000370,5000380,5000410, 5000420,5000430,5000440,5000450,5000460, 5000470,5000480,5000510,5000525,5000530, 5000540,5000550,5000560,5000570,5000580)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/ptc00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpPtc.yaml --register-dataset-types butler certify-calibrations ${REPO} u/czw/ptc00 LSSTCam-imSim/calib/dp0 --begin-date 1980 - 01 - 01 --end-date 2050 - 01 - 01 ptc   pipetask -- long -log run -d " exposure IN (5000110)" -b ${REPO} -i LSSTCam-imSim/raw/all,LSSTCam-imSim/calib,LSSTCam-imSim/calib/dp0 -o u/czw/bfk00 -p ${OBS_LSST_DIR}/pipelines/imsim/cpBfkSolve.yaml --register-dataset-types  
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Just in case you missed it you need a $REPO in the write-curated-calibrations line.

            Show
            tjenness Tim Jenness added a comment - Just in case you missed it you need a $REPO in the write-curated-calibrations line.
            Hide
            Permalink
            dtaranu Dan Taranu added a comment -

            Looks good. I was able to run all of the steps up to the second butler-calibrations call, which I stopped after realizing how long it was going to take. I presume the rest would have finished normally.

            Show
            dtaranu Dan Taranu added a comment - Looks good. I was able to run all of the steps up to the second butler-calibrations call, which I stopped after realizing how long it was going to take. I presume the rest would have finished normally.

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              czw Christopher Waters
              Reviewers:
              Dan Taranu
              Watchers:
              Christopher Waters, Dan Taranu, James Chiang, Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.