Show
added a comment - - edited The PTC code has been split into two tasks: extract and solve . The first task receives as input a list of flats, and sorts these flats in pairs taken at the same time (if there’s a different number of flats, those flats are discarded). The mean, variance (using afw), and covariances (as in Astier+19) are measured from the difference of the flat pairs at a given time. The measured covariances at that time (along with other quantities such as the mean) are stored in a PTC dataset object ( PhotonTransferCurveDataset ), which gets partially filled. The number of partially-filled PTC dataset objects will be less than the number of input exposures, but gen3 requires/assumes that the number of input dimensions matches bijectively the number of output dimensions. Therefore, a number of “dummy” PTC dataset are inserted in the output list that has the partially-filled PTC datasets with the covariances.
The solve task assembles all the individual PTC datasets into a final PTC dataset. Then, it fits the measured (co)variances to a polynomial model or to the models described in equations 16 and 20 of Astier+19 (referred to as POLYNOMIAL , EXPAPPROXIMATION , and FULLCOVARIANCE in the configuration options of the solve task, respectively).
The doPhotodiode option (to calibrate signal-chain non-linearity) is currently implemented for gen2 only.
Example of constructing a gen3 repo and running the PTC task on BOT data, detector 94 (in /project/plazas/WORK/ DM-23159 /TEST_BOT with this branch and w_2020_52 ):
butler create TEST_BOT
butler register-instrument TEST_BOT/ lsst.obs.lsst.LsstCam
butler write-curated-calibrations TEST_BOT/ lsst.obs.lsst.LsstCam
for i in $(ls -d /project/shared/BOT/_parent/raw/2020-10-08/*/); butler ingest-raws TEST_BOT/ $i; done
Bias:
pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800090, 3020100800091, 3020100800092, 3020100800093, 3020100800094) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib -o test_bias2 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpBias.yaml -c isr:doDefect=False --register-dataset-types
butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_bias2 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 bias
Dark:
pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800045) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_dark6 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpDark.yaml -c isr:doDefect=False -c isr:doCrosstalk=False -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 --register-dataset-types
butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_dark6 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 dark
Flat:
pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800140, 3020100800141) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_flat5 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpFlat.yaml -c isr:doDefect=False -c isr:doCrosstalk=False -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 -c cpFlatMeasure:doVignette=False --register-dataset-types
butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_flat5 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 flat
Defects:
pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800140,3020100800141,3020100800145) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_defects -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/findDefects.yaml -c measureDefects:badOnAndOffPixelColumnThreshold=50 --register-dataset-types
butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_defects calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 defects
ISR:
pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o ptcTestPostIsr.02 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/isrForPtc.yaml -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 -c isr:doFringe=False -c isr:doSuspect=True -c isr:edgeMaskLevel=AMP -c isr:numEdgeSuspect=10 -c isr:doFlat=False --register-dataset-types
with /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/isrForPtc.yaml
description: cp_pipe PTC calibration construction.
tasks:
isr:
class: lsst.ip.isr.isrTask.IsrTask
config:
connections.ccdExposure: 'raw'
connections.outputExposure: 'cpPtcProc'
doWrite: True
doOverscan: True
doAssembleCcd: True
doBias: True
doVariance: True
doLinearize: False
doCrosstalk: False
doBrighterFatter: False
doDark: True
doStrayLight: False
doFlat: True
doFringe: False
doApplyGains: False
doDefect: True
doNanMasking: True
doInterpolate: False
doSaturation: False
doSaturationInterpolation: False
growSaturationFootprintSize: 0
PTC (for some reason it could not find the flat, so I set it to False):
pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestResultado.06 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION --register-dataset-types
(with ISR options: )
pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestProbando.04 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c isr:doFlat=False -c isr:doCrosstalk=False --register-dataset-types
Output dataset: ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits
from lsst.ip.isr import PhotonTransferCurveDataset
filename = "ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits"
fromFits = PhotonTransferCurveDataset.readFits(filename)
fromFits.gain
fromFits.noise
Gains:
{'C00': 1.062342253140052,
'C01': 1.069284400977535,
'C02': 1.0718946810066032,
'C03': 1.0698720790589018,
'C04': 1.0755230961249225,
'C05': 1.0697500124030388,
'C06': 1.0696513340670983,
'C07': 1.0618395593772259,
'C10': 1.0788131187964267,
'C11': 1.0833694635298263,
'C12': 1.095240826031433,
'C13': 1.0914294799719313,
'C14': 1.0958563288441774,
'C15': 1.0985139310664527,
'C16': 1.0973343675070544,
'C17': 1.0969255452082705}
Noise:
{'C00': 6.151925075768491,
'C01': 5.97459431206064,
'C02': 5.6993306138734114,
'C03': 5.621638005069704,
'C04': 5.6358216929068226,
'C05': 6.082843390559317,
'C06': 5.569225331081291,
'C07': 6.158968155376583,
'C10': 5.876614760829971,
'C11': 6.036380486530242,
'C12': 5.772919322451841,
'C13': 5.896424344627466,
'C14': 6.068517828241981,
'C15': 6.7266935200050675,
'C16': 6.674482599560533,
'C17': 5.951403680148852}
Using FULLCOVARIANCE : ptcTestResultado.07/20201231T024110Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_07_20201231T024110Z.fits
Gain:
{'C00': 1.0425526047034526,
'C01': 1.0530311663384373,
'C02': 1.0540685272603145,
'C03': 1.0386811075221716,
'C04': 1.061475832585829,
'C05': 1.0555755173001389,
'C06': 1.0434694202440942,
'C07': 1.0356749845692936,
'C10': 1.0582041818437944,
'C11': 1.0645796838195933,
'C12': 1.0819382322040194,
'C13': 1.0939012841298579,
'C14': 1.0848968736492128,
'C15': 1.078412587502557,
'C16': 1.0788399491643776,
'C17': 1.0824094906159403}
Noise:
{'C00': 5.033597545607389,
'C01': 5.1658357180823,
'C02': 4.7660533194003785,
'C03': 4.077912281257251,
'C04': 4.852569375938531,
'C05': 5.323858607073495,
'C06': 4.4042938592755725,
'C07': 5.685987684065248,
'C10': 4.857324917128721,
'C11': 5.2376226461227455,
'C12': 5.9238080874005545,
'C13': 6.717944248640032,
'C14': 5.572157959987069,
'C15': 6.026442101939837,
'C16': 5.9025554026867795,
'C17': 5.471607360413247}
Suggestion by C. Waters: