[DM-23159] Make PTC task compatible with RFC-352 - Jira

Andrés Alejandro Plazas Malagón created issue - 23/Jan/20 3:02 AM

Andrés Alejandro Plazas Malagón made changes - 23/Jan/20 3:02 AM

Field	Original Value	New Value
Epic Link		~~DM-22594~~ [ 427662 ]

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 23/Jan/20 3:06 AM - edited

Suggestion by C. Waters:

Move the dataset into a new run() method, that expects a list of ISR processed exposures, and returns a struct containing all of the output products that will be written to disk.
L328-337 (https://github.com/lsst/cp_pipe/blob/tickets/DM-21221/python/lsst/cp/pipe/ptc.py) stay in runDataRef, as they're doing the ISR processing to generate the postISRCCD files that will be passed to run.

L339-356 move to a run method, which is the "algorithmic" section. The idea being that run( list(postISRCCD), config) yields Struct(plots=list(ptcFitPlots), ptcDataset=dataset, linearizer=LinearizerLookupTable, etcDataToSave=somethingElse) , which runDataRef then calls butler.put on.

Show

Andrés Alejandro Plazas Malagón added a comment - 23/Jan/20 3:06 AM - edited Suggestion by C. Waters: Move the dataset into a new run() method, that expects a list of ISR processed exposures, and returns a struct containing all of the output products that will be written to disk. L328-337 ( https://github.com/lsst/cp_pipe/blob/tickets/DM-21221/python/lsst/cp/pipe/ptc.py ) stay in runDataRef, as they're doing the ISR processing to generate the postISRCCD files that will be passed to run. L339-356 move to a run method, which is the "algorithmic" section. The idea being that run( list(postISRCCD), config) yields Struct(plots=list(ptcFitPlots), ptcDataset=dataset, linearizer=LinearizerLookupTable, etcDataToSave=somethingElse) , which runDataRef then calls butler.put on.

Yusra AlSayyad made changes - 24/Mar/20 10:45 PM

Epic Link

~~DM-22594~~ [ 427662 ]

DM-23745 [ 431402 ]

Christopher Waters made changes - 12/Oct/20 2:57 PM

Epic Link

DM-23745 [ 431402 ]

~~DM-26805~~ [ 439757 ]

Christopher Waters made changes - 12/Oct/20 2:57 PM

Urgent?

off

Christopher Waters made changes - 12/Oct/20 3:13 PM

Remote Link

This issue links to "Page (Confluence)" [ 26132 ]

Christopher Waters made changes - 15/Oct/20 8:13 PM

Link

This issue blocks ~~DM-21786~~ [ ~~DM-21786~~ ]

Andrés Alejandro Plazas Malagón made changes - 19/Oct/20 3:29 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Christopher Waters made changes - 19/Oct/20 6:32 PM

Remote Link

This issue links to "Page (Confluence)" [ 26194 ]

Christopher Waters made changes - 26/Oct/20 4:26 PM

Remote Link

This issue links to "Page (Confluence)" [ 26320 ]

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 27/Oct/20 9:16 PM - edited

ci_cpp_gen2 build (w_2020_43 setup): ptc_ci_cpp_gen2_2020OCT27.txt

Show

Andrés Alejandro Plazas Malagón added a comment - 27/Oct/20 9:16 PM - edited ci_cpp_gen2 build ( w_2020_43 setup ): ptc_ci_cpp_gen2_2020OCT27.txt

Andrés Alejandro Plazas Malagón made changes - 27/Oct/20 9:16 PM

Attachment

ptc_ci_cpp_gen2_2020OCT27.txt [ 45988 ]

Christopher Waters made changes - 30/Oct/20 6:04 PM

Link

This issue blocks DM-27385 [ DM-27385 ]

Andrés Alejandro Plazas Malagón made changes - 04/Nov/20 2:24 AM

Link

This issue is blocked by ~~DM-27419~~ [ ~~DM-27419~~ ]

Christopher Waters made changes - 10/Dec/20 5:37 PM

Epic Link

~~DM-26805~~ [ 439757 ]

~~DM-27953~~ [ 442726 ]

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 31/Dec/20 2:35 AM - edited

The PTC code has been split into two tasks: extract and solve. The first task receives as input a list of flats, and sorts these flats in pairs taken at the same time (if there’s a different number of flats, those flats are discarded). The mean, variance (using afw), and covariances (as in Astier+19) are measured from the difference of the flat pairs at a given time. The measured covariances at that time (along with other quantities such as the mean) are stored in a PTC dataset object (PhotonTransferCurveDataset), which gets partially filled. The number of partially-filled PTC dataset objects will be less than the number of input exposures, but gen3 requires/assumes that the number of input dimensions matches bijectively the number of output dimensions. Therefore, a number of “dummy” PTC dataset are inserted in the output list that has the partially-filled PTC datasets with the covariances.

The solve task assembles all the individual PTC datasets into a final PTC dataset. Then, it fits the measured (co)variances to a polynomial model or to the models described in equations 16 and 20 of Astier+19 (referred to as POLYNOMIAL,EXPAPPROXIMATION, and FULLCOVARIANCE in the configuration options of the solve task, respectively).

The doPhotodiode option (to calibrate signal-chain non-linearity) is currently implemented for gen2 only.

Example of constructing a gen3 repo and running the PTC task on BOT data, detector 94 (in /project/plazas/WORK/DM-23159/TEST_BOT with this branch and w_2020_52):

 butler create TEST_BOT

 butler register-instrument TEST_BOT/ lsst.obs.lsst.LsstCam

 butler write-curated-calibrations TEST_BOT/ lsst.obs.lsst.LsstCam

for i in $(ls -d /project/shared/BOT/_parent/raw/2020-10-08/*/); butler ingest-raws TEST_BOT/ $i; done

Bias:

pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800090, 3020100800091, 3020100800092, 3020100800093, 3020100800094) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib -o test_bias2 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpBias.yaml -c isr:doDefect=False --register-dataset-types

butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_bias2 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 bias

Dark:

pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800045) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_dark6 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpDark.yaml -c isr:doDefect=False -c isr:doCrosstalk=False -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 --register-dataset-types

butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_dark6 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 dark

Flat:

pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800140, 3020100800141) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_flat5 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpFlat.yaml -c isr:doDefect=False -c isr:doCrosstalk=False -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 -c cpFlatMeasure:doVignette=False --register-dataset-types

butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_flat5 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 flat

Defects:

pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800140,3020100800141,3020100800145) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_defects -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/findDefects.yaml  -c measureDefects:badOnAndOffPixelColumnThreshold=50 --register-dataset-types

butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_defects calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 defects

ISR:

pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o ptcTestPostIsr.02 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/isrForPtc.yaml -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 -c isr:doFringe=False -c isr:doSuspect=True -c isr:edgeMaskLevel=AMP -c isr:numEdgeSuspect=10 -c isr:doFlat=False --register-dataset-types

with /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/isrForPtc.yaml

description: cp_pipe PTC calibration construction.

tasks:

  isr:

    class: lsst.ip.isr.isrTask.IsrTask

    config:

      connections.ccdExposure: 'raw'

      connections.outputExposure: 'cpPtcProc'

      doWrite: True

      doOverscan: True

      doAssembleCcd: True

      doBias: True

      doVariance: True

      doLinearize: False

      doCrosstalk: False

      doBrighterFatter: False

      doDark: True

      doStrayLight: False

      doFlat: True

      doFringe: False

      doApplyGains: False

      doDefect: True

      doNanMasking: True

      doInterpolate: False

      doSaturation: False

      doSaturationInterpolation: False

      growSaturationFootprintSize: 0

PTC (for some reason it could not find the flat, so I set it to False):

pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestResultado.06 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION --register-dataset-types

(with ISR options: )

pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestProbando.04 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c isr:doFlat=False -c isr:doCrosstalk=False --register-dataset-types

Output dataset: ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits

from lsst.ip.isr import PhotonTransferCurveDataset

filename="ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits"

fromFits = PhotonTransferCurveDataset.readFits(filename)

fromFits.gain

fromFits.noise

Gains:

{'C00': 1.062342253140052,

 'C01': 1.069284400977535,

 'C02': 1.0718946810066032,

 'C03': 1.0698720790589018,

 'C04': 1.0755230961249225,

 'C05': 1.0697500124030388,

 'C06': 1.0696513340670983,

 'C07': 1.0618395593772259,

 'C10': 1.0788131187964267,

 'C11': 1.0833694635298263,

 'C12': 1.095240826031433,

 'C13': 1.0914294799719313,

 'C14': 1.0958563288441774,

 'C15': 1.0985139310664527,

 'C16': 1.0973343675070544,

 'C17': 1.0969255452082705}

Noise:

{'C00': 6.151925075768491,

 'C01': 5.97459431206064,

 'C02': 5.6993306138734114,

 'C03': 5.621638005069704,

 'C04': 5.6358216929068226,

 'C05': 6.082843390559317,

 'C06': 5.569225331081291,

 'C07': 6.158968155376583,

 'C10': 5.876614760829971,

 'C11': 6.036380486530242,

 'C12': 5.772919322451841,

 'C13': 5.896424344627466,

 'C14': 6.068517828241981,

 'C15': 6.7266935200050675,

 'C16': 6.674482599560533,

 'C17': 5.951403680148852}

Using FULLCOVARIANCE: ptcTestResultado.07/20201231T024110Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_07_20201231T024110Z.fits

Gain:

{'C00': 1.0425526047034526,

 'C01': 1.0530311663384373,

 'C02': 1.0540685272603145,

 'C03': 1.0386811075221716,

 'C04': 1.061475832585829,

 'C05': 1.0555755173001389,

 'C06': 1.0434694202440942,

 'C07': 1.0356749845692936,

 'C10': 1.0582041818437944,

 'C11': 1.0645796838195933,

 'C12': 1.0819382322040194,

 'C13': 1.0939012841298579,

 'C14': 1.0848968736492128,

 'C15': 1.078412587502557,

 'C16': 1.0788399491643776,

 'C17': 1.0824094906159403}

Noise:

{'C00': 5.033597545607389,

 'C01': 5.1658357180823,

 'C02': 4.7660533194003785,

 'C03': 4.077912281257251,

 'C04': 4.852569375938531,

 'C05': 5.323858607073495,

 'C06': 4.4042938592755725,

 'C07': 5.685987684065248,

 'C10': 4.857324917128721,

 'C11': 5.2376226461227455,

 'C12': 5.9238080874005545,

 'C13': 6.717944248640032,

 'C14': 5.572157959987069,

 'C15': 6.026442101939837,

 'C16': 5.9025554026867795,

 'C17': 5.471607360413247}

Show

Andrés Alejandro Plazas Malagón added a comment - 31/Dec/20 2:35 AM - edited The PTC code has been split into two tasks: extract and solve . The first task receives as input a list of flats, and sorts these flats in pairs taken at the same time (if there’s a different number of flats, those flats are discarded). The mean, variance (using afw), and covariances (as in Astier+19) are measured from the difference of the flat pairs at a given time. The measured covariances at that time (along with other quantities such as the mean) are stored in a PTC dataset object ( PhotonTransferCurveDataset ), which gets partially filled. The number of partially-filled PTC dataset objects will be less than the number of input exposures, but gen3 requires/assumes that the number of input dimensions matches bijectively the number of output dimensions. Therefore, a number of “dummy” PTC dataset are inserted in the output list that has the partially-filled PTC datasets with the covariances. The solve task assembles all the individual PTC datasets into a final PTC dataset. Then, it fits the measured (co)variances to a polynomial model or to the models described in equations 16 and 20 of Astier+19 (referred to as POLYNOMIAL , EXPAPPROXIMATION , and FULLCOVARIANCE in the configuration options of the solve task, respectively). The doPhotodiode option (to calibrate signal-chain non-linearity) is currently implemented for gen2 only. Example of constructing a gen3 repo and running the PTC task on BOT data, detector 94 (in /project/plazas/WORK/ DM-23159 /TEST_BOT with this branch and w_2020_52 ): butler create TEST_BOT butler register-instrument TEST_BOT/ lsst.obs.lsst.LsstCam butler write-curated-calibrations TEST_BOT/ lsst.obs.lsst.LsstCam for i in $(ls -d /project/shared/BOT/_parent/raw/2020-10-08/*/); butler ingest-raws TEST_BOT/ $i; done Bias: pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800090, 3020100800091, 3020100800092, 3020100800093, 3020100800094) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib -o test_bias2 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpBias.yaml -c isr:doDefect=False --register-dataset-types butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_bias2 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 bias Dark: pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800045) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_dark6 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpDark.yaml -c isr:doDefect=False -c isr:doCrosstalk=False -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 --register-dataset-types butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_dark6 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 dark Flat: pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800140, 3020100800141) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_flat5 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/cpFlat.yaml -c isr:doDefect=False -c isr:doCrosstalk=False -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 -c cpFlatMeasure:doVignette=False --register-dataset-types butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_flat5 calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 flat Defects: pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800140,3020100800141,3020100800145) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o test_defects -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/findDefects.yaml -c measureDefects:badOnAndOffPixelColumnThreshold=50 --register-dataset-types butler certify-calibrations /project/plazas/WORK/DM-23159/TEST_BOT test_defects calib/v00 --begin-date 1980-01-01 --end-date 2050-01-01 defects ISR: pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i LSSTCam/raw/all,LSSTCam/calib,calib/v00 -o ptcTestPostIsr.02 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/isrForPtc.yaml -c isr:overscan.fitType=MEDIAN -c isr:overscan.order=1 -c isr:doFringe=False -c isr:doSuspect=True -c isr:edgeMaskLevel=AMP -c isr:numEdgeSuspect=10 -c isr:doFlat=False --register-dataset-types with /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/isrForPtc.yaml description: cp_pipe PTC calibration construction. tasks: isr: class: lsst.ip.isr.isrTask.IsrTask config: connections.ccdExposure: 'raw' connections.outputExposure: 'cpPtcProc' doWrite: True doOverscan: True doAssembleCcd: True doBias: True doVariance: True doLinearize: False doCrosstalk: False doBrighterFatter: False doDark: True doStrayLight: False doFlat: True doFringe: False doApplyGains: False doDefect: True doNanMasking: True doInterpolate: False doSaturation: False doSaturationInterpolation: False growSaturationFootprintSize: 0 PTC (for some reason it could not find the flat, so I set it to False): pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestResultado.06 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION --register-dataset-types (with ISR options: ) pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestProbando.04 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION -c isr:doFlat=False -c isr:doCrosstalk=False --register-dataset-types Output dataset: ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits from lsst.ip.isr import PhotonTransferCurveDataset filename = "ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits" fromFits = PhotonTransferCurveDataset.readFits(filename) fromFits.gain fromFits.noise Gains: {'C00': 1.062342253140052, 'C01': 1.069284400977535, 'C02': 1.0718946810066032, 'C03': 1.0698720790589018, 'C04': 1.0755230961249225, 'C05': 1.0697500124030388, 'C06': 1.0696513340670983, 'C07': 1.0618395593772259, 'C10': 1.0788131187964267, 'C11': 1.0833694635298263, 'C12': 1.095240826031433, 'C13': 1.0914294799719313, 'C14': 1.0958563288441774, 'C15': 1.0985139310664527, 'C16': 1.0973343675070544, 'C17': 1.0969255452082705} Noise: {'C00': 6.151925075768491, 'C01': 5.97459431206064, 'C02': 5.6993306138734114, 'C03': 5.621638005069704, 'C04': 5.6358216929068226, 'C05': 6.082843390559317, 'C06': 5.569225331081291, 'C07': 6.158968155376583, 'C10': 5.876614760829971, 'C11': 6.036380486530242, 'C12': 5.772919322451841, 'C13': 5.896424344627466, 'C14': 6.068517828241981, 'C15': 6.7266935200050675, 'C16': 6.674482599560533, 'C17': 5.951403680148852} Using FULLCOVARIANCE : ptcTestResultado.07/20201231T024110Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_07_20201231T024110Z.fits Gain: {'C00': 1.0425526047034526, 'C01': 1.0530311663384373, 'C02': 1.0540685272603145, 'C03': 1.0386811075221716, 'C04': 1.061475832585829, 'C05': 1.0555755173001389, 'C06': 1.0434694202440942, 'C07': 1.0356749845692936, 'C10': 1.0582041818437944, 'C11': 1.0645796838195933, 'C12': 1.0819382322040194, 'C13': 1.0939012841298579, 'C14': 1.0848968736492128, 'C15': 1.078412587502557, 'C16': 1.0788399491643776, 'C17': 1.0824094906159403} Noise: {'C00': 5.033597545607389, 'C01': 5.1658357180823, 'C02': 4.7660533194003785, 'C03': 4.077912281257251, 'C04': 4.852569375938531, 'C05': 5.323858607073495, 'C06': 4.4042938592755725, 'C07': 5.685987684065248, 'C10': 4.857324917128721, 'C11': 5.2376226461227455, 'C12': 5.9238080874005545, 'C13': 6.717944248640032, 'C14': 5.572157959987069, 'C15': 6.026442101939837, 'C16': 5.9025554026867795, 'C17': 5.471607360413247}

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 02/Jan/21 4:51 PM - edited

Comparing the results in the comment above for the same detector with the gen2 command in this branch (~~DM-23159~~):

measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-23159/2021JAN02 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c solve.ptcFitType=EXPAPPROXIMATION doPhotodiode=False --clobber-config --clobber-version -j 1

Using EXPAPPROXIMATION: /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-23159/2021JAN02/calibrations/ptc/ptcDataset-det094.fits

Gain:

{'C00': 1.0623665682015864,

 'C01': 1.0692736458129461,

 'C02': 1.0719061777555439,

 'C03': 1.0698186646485832,

 'C04': 1.0755065917205915,

 'C05': 1.0697223373796645,

 'C06': 1.0696511819243888,

 'C07': 1.0618560986812993,

 'C10': 1.0788291305881819,

 'C11': 1.0833586774001116,

 'C12': 1.0952379465349606,

 'C13': 1.0914619568425337,

 'C14': 1.0958556943098245,

 'C15': 1.0984705558827348,

 'C16': 1.0973459162125423,

 'C17': 1.096906929928102}

Noise:

{'C00': 6.153475805831076,

 'C01': 5.973331468670271,

 'C02': 5.699858878211565,

 'C03': 5.61691772662747,

 'C04': 5.635055718329871,

 'C05': 6.079625027880279,

 'C06': 5.569378782470496,

 'C07': 6.159928134084513,

 'C10': 5.875580728681517,

 'C11': 6.0344819056114,

 'C12': 5.772831136821642,

 'C13': 5.899530865591045,

 'C14': 6.069236296852714,

 'C15': 6.724938160329195,

 'C16': 6.67640273657015,

 'C17': 5.950307038961287}

Using FULLCOVARIANCE: /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-23159/2021JAN02/FULLCOVARIANCE/calibrations/ptc/ptcDataset-det094.fits

Gain:

{'C00': 1.0425531053098844,

 'C01': 1.0530460028237154,

 'C02': 1.0540841995693877,

 'C03': 1.0386628419980093,

 'C04': 1.061489431439738,

 'C05': 1.0556060905559954,

 'C06': 1.0434438508943065,

 'C07': 1.0356673273507941,

 'C10': 1.0582499068756281,

 'C11': 1.0645580775004337,

 'C12': 1.081944596613428,

 'C13': 1.0939098534071015,

 'C14': 1.084900009252566,

 'C15': 1.0783756852548805,

 'C16': 1.0788008311368777,

 'C17': 1.0809517945824252}

Noise:

{'C00': 5.033844945221069,

 'C01': 5.16586296071402,

 'C02': 4.767118832864183,

 'C03': 4.076103010864511,

 'C04': 4.853851376347369,

 'C05': 5.324508253437716,

 'C06': 4.403355871485114,

 'C07': 5.684202672212702,

 'C10': 4.859855629246495,

 'C11': 5.235422775870591,

 'C12': 5.925012945621021,

 'C13': 6.718464542041558,

 'C14': 5.5733037153519325,

 'C15': 6.025369660453583,

 'C16': 5.901638560696901,

 'C17': 5.429647760406786}

Show

Andrés Alejandro Plazas Malagón added a comment - 02/Jan/21 4:51 PM - edited Comparing the results in the comment above for the same detector with the gen2 command in this branch ( DM-23159 ): measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-23159/2021JAN02 --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c solve.ptcFitType=EXPAPPROXIMATION doPhotodiode=False --clobber-config --clobber-version -j 1 Using EXPAPPROXIMATION : /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/ DM-23159 /2021JAN02/calibrations/ptc/ptcDataset-det094.fits Gain: {'C00': 1.0623665682015864, 'C01': 1.0692736458129461, 'C02': 1.0719061777555439, 'C03': 1.0698186646485832, 'C04': 1.0755065917205915, 'C05': 1.0697223373796645, 'C06': 1.0696511819243888, 'C07': 1.0618560986812993, 'C10': 1.0788291305881819, 'C11': 1.0833586774001116, 'C12': 1.0952379465349606, 'C13': 1.0914619568425337, 'C14': 1.0958556943098245, 'C15': 1.0984705558827348, 'C16': 1.0973459162125423, 'C17': 1.096906929928102} Noise: {'C00': 6.153475805831076, 'C01': 5.973331468670271, 'C02': 5.699858878211565, 'C03': 5.61691772662747, 'C04': 5.635055718329871, 'C05': 6.079625027880279, 'C06': 5.569378782470496, 'C07': 6.159928134084513, 'C10': 5.875580728681517, 'C11': 6.0344819056114, 'C12': 5.772831136821642, 'C13': 5.899530865591045, 'C14': 6.069236296852714, 'C15': 6.724938160329195, 'C16': 6.67640273657015, 'C17': 5.950307038961287} Using FULLCOVARIANCE : /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/ DM-23159 /2021JAN02/FULLCOVARIANCE/calibrations/ptc/ptcDataset-det094.fits Gain: {'C00': 1.0425531053098844, 'C01': 1.0530460028237154, 'C02': 1.0540841995693877, 'C03': 1.0386628419980093, 'C04': 1.061489431439738, 'C05': 1.0556060905559954, 'C06': 1.0434438508943065, 'C07': 1.0356673273507941, 'C10': 1.0582499068756281, 'C11': 1.0645580775004337, 'C12': 1.081944596613428, 'C13': 1.0939098534071015, 'C14': 1.084900009252566, 'C15': 1.0783756852548805, 'C16': 1.0788008311368777, 'C17': 1.0809517945824252} Noise: {'C00': 5.033844945221069, 'C01': 5.16586296071402, 'C02': 4.767118832864183, 'C03': 4.076103010864511, 'C04': 4.853851376347369, 'C05': 5.324508253437716, 'C06': 4.403355871485114, 'C07': 5.684202672212702, 'C10': 4.859855629246495, 'C11': 5.235422775870591, 'C12': 5.925012945621021, 'C13': 6.718464542041558, 'C14': 5.5733037153519325, 'C15': 6.025369660453583, 'C16': 5.901638560696901, 'C17': 5.429647760406786}

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 02/Jan/21 10:18 PM

Comparing the results to the verions of cp_pipe and ip_isr currently in master (w_2020_52):

ptcFitType: EXPAPPROXIMATION: /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-23159/2021JAN02/EXPAPPROX-MASTER/calibrations/ptc/ptcDataset-det094.fits

/home/plazas/lsst_devel/LSST/cp_pipe/bin/measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-23159/2021JAN02/EXPAPPROX-MASTER --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c ptcFitType=EXPAPPROXIMATION doPhotodiode=False --clobber-config --clobber-version -j 1

Gain:

{'C00':  1.0623665682015864

'C01':  1.0692736458129461

'C02':  1.0719061777555439

'C03':  1.0698186646485832

'C04':  1.0755065917205915

'C05':  1.0697223373796645

'C06':  1.0696511819243888

'C07':  1.0618560986812993

'C10':  1.0788291305881819

'C11':  1.0833586774001116

'C12':  1.0952379465349606

'C13':  1.0914619568425337

'C14':  1.0958556943098245

'C15':  1.0984705558827348

'C16':  1.0973459162125423

'C17':  1.096906929928102}

Noise:

'C00':  6.153475805831076

'C01':  5.973331468670271

'C02':  5.699858878211565

'C03':  5.61691772662747

'C04':  5.635055718329871

'C05':  6.079625027880279

'C06':  5.569378782470496

'C07':  6.159928134084513

'C10':  5.875580728681517

'C11':  6.0344819056114

'C12':  5.772831136821642

'C13':  5.899530865591045

'C14':  6.069236296852714

'C15':  6.724938160329195

'C16':  6.67640273657015

'C17':  5.950307038961287

ptcFitType: FULLCOVARIANCE:

/project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/DM-23159/2021JAN02/EXPAPPROX-FULLCOVARIANCE/calibrations/ptc/ptcDataset-det094.fits

measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-23159/2021JAN02/EXPAPPROX-FULLCOVARIANCE --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c ptcFitType=FULLCOVARIANCE doPhotodiode=False --clobber-config --clobber-version -j 1

Gain:

'C00':  1.0425556887637621

'C01':  1.0530442420728823

'C02':  1.0540860068259987

'C03':  1.0386621624006385

'C04':  1.0614891977940373

'C05':  1.0556061121455635

'C06':  1.0434440018245594

'C07':  1.035667225196643

'C10':  1.058252331030148

'C11':  1.0645580285748109

'C12':  1.0819436197242243

'C13':  1.0939079018240114

'C14':  1.0849000245608877

'C15':  1.07837551226726

'C16':  1.078800800817564

'C17':  1.0809518420287751

Noise:

'C00':  5.033950113017649

'C01':  5.165794683782765

'C02':  4.767193068143683

'C03':  4.076072653277696

'C04':  4.853841940191721

'C05':  5.3245091460021134

'C06':  4.403362155458655

'C07':  5.684199300487573

'C10':  4.859952630497146

'C11':  5.235420893116862

'C12':  5.9249790705589565

'C13':  6.7183983328087855

'C14':  5.573304230848151

'C15':  6.0253641613470155

'C16':  5.901637552758507

'C17':  5.429649457878611

Show

Andrés Alejandro Plazas Malagón added a comment - 02/Jan/21 10:18 PM Comparing the results to the verions of cp_pipe and ip_isr currently in master ( w_2020_52 ): ptcFitType: EXPAPPROXIMATION : /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/ DM-23159 /2021JAN02/EXPAPPROX-MASTER/calibrations/ptc/ptcDataset-det094.fits /home/plazas/lsst_devel/LSST/cp_pipe/bin/measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-23159/2021JAN02/EXPAPPROX-MASTER --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c ptcFitType=EXPAPPROXIMATION doPhotodiode=False --clobber-config --clobber-version -j 1 Gain: {'C00': 1.0623665682015864 'C01': 1.0692736458129461 'C02': 1.0719061777555439 'C03': 1.0698186646485832 'C04': 1.0755065917205915 'C05': 1.0697223373796645 'C06': 1.0696511819243888 'C07': 1.0618560986812993 'C10': 1.0788291305881819 'C11': 1.0833586774001116 'C12': 1.0952379465349606 'C13': 1.0914619568425337 'C14': 1.0958556943098245 'C15': 1.0984705558827348 'C16': 1.0973459162125423 'C17': 1.096906929928102} Noise: 'C00': 6.153475805831076 'C01': 5.973331468670271 'C02': 5.699858878211565 'C03': 5.61691772662747 'C04': 5.635055718329871 'C05': 6.079625027880279 'C06': 5.569378782470496 'C07': 6.159928134084513 'C10': 5.875580728681517 'C11': 6.0344819056114 'C12': 5.772831136821642 'C13': 5.899530865591045 'C14': 6.069236296852714 'C15': 6.724938160329195 'C16': 6.67640273657015 'C17': 5.950307038961287 ptcFitType: FULLCOVARIANCE : /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/rerun/plazas/ DM-23159 /2021JAN02/EXPAPPROX-FULLCOVARIANCE/calibrations/ptc/ptcDataset-det094.fits measurePhotonTransferCurve.py /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606 --rerun plazas/DM-23159/2021JAN02/EXPAPPROX-FULLCOVARIANCE --id detector=94 expId=3020100800155^3020100800156^3020100800158^3020100800159^3020100800185^3020100800186^3020100800161^3020100800162^3020100800188^3020100800189^3020100800164^3020100800165^3020100800191^3020100800192^3020100800167^3020100800168^3020100800194^3020100800195^3020100800170^3020100800171^3020100800197^3020100800198^3020100800173^3020100800174^3020100800200^3020100800201^3020100800176^3020100800177^3020100800203^3020100800204^3020100800179^3020100800180^3020100800206^3020100800207^3020100800182^3020100800183^3020100800209^3020100800210^3020100800212^3020100800213^3020100800215^3020100800216 -c ptcFitType=FULLCOVARIANCE doPhotodiode=False --clobber-config --clobber-version -j 1 Gain: 'C00': 1.0425556887637621 'C01': 1.0530442420728823 'C02': 1.0540860068259987 'C03': 1.0386621624006385 'C04': 1.0614891977940373 'C05': 1.0556061121455635 'C06': 1.0434440018245594 'C07': 1.035667225196643 'C10': 1.058252331030148 'C11': 1.0645580285748109 'C12': 1.0819436197242243 'C13': 1.0939079018240114 'C14': 1.0849000245608877 'C15': 1.07837551226726 'C16': 1.078800800817564 'C17': 1.0809518420287751 Noise: 'C00': 5.033950113017649 'C01': 5.165794683782765 'C02': 4.767193068143683 'C03': 4.076072653277696 'C04': 4.853841940191721 'C05': 5.3245091460021134 'C06': 4.403362155458655 'C07': 5.684199300487573 'C10': 4.859952630497146 'C11': 5.235420893116862 'C12': 5.9249790705589565 'C13': 6.7183983328087855 'C14': 5.573304230848151 'C15': 6.0253641613470155 'C16': 5.901637552758507 'C17': 5.429649457878611

Andrés Alejandro Plazas Malagón made changes - 06/Jan/21 5:54 PM

Attachment

PTC_det94_EXPAPPROX_DM-23159.pdf [ 47066 ]

Andrés Alejandro Plazas Malagón made changes - 06/Jan/21 6:01 PM

Attachment

PTC_det94_FULLCOVARIANCE_DM-23159.pdf [ 47067 ]

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 06/Jan/21 6:01 PM - edited

Plotting routine:

The gen2 plotting code currently in master has been moved to cp_pipepython/lsst/cp/pipe/plotPtcGen2.py to maintain backward compatibility with gen2; most of that code remains in cp_pipe/python/lsst/cp/pipe/plotPtc.py and has been wrapped around a script in cp_pipe/python/lsst/cp/pipe/script/plot_ptc_dataset.py.

The plotting code can now be called as follows:

plotPhotonTransferCurve.py /project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits --detNum=94 --outDir=/project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.06/PDF_PLOTS

PDF (EXPAPPROXIMATION with gen3 code): PTC_det94_EXPAPPROX_DM-23159.pdf

plotPhotonTransferCurve.py /project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.07/20201231T024110Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_07_20201231T024110Z.fits --detNum=94 --outDir=/project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.06/PDF_PLOTS_FULLCOVARIANCE

PDF (FULLCOVARIANCE with gen3 code): PTC_det94_FULLCOVARIANCE_DM-23159.pdf

Show

Andrés Alejandro Plazas Malagón added a comment - 06/Jan/21 6:01 PM - edited Plotting routine: The gen2 plotting code currently in master has been moved to cp_pipepython/lsst/cp/pipe/plotPtcGen2.py to maintain backward compatibility with gen2; most of that code remains in cp_pipe/python/lsst/cp/pipe/plotPtc.py and has been wrapped around a script in cp_pipe/python/lsst/cp/pipe/script/plot_ptc_dataset.py . The plotting code can now be called as follows: plotPhotonTransferCurve.py /project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.06/20201231T020616Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_06_20201231T020616Z.fits --detNum=94 --outDir=/project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.06/PDF_PLOTS PDF ( EXPAPPROXIMATION with gen3 code): PTC_det94_EXPAPPROX_DM-23159.pdf plotPhotonTransferCurve.py /project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.07/20201231T024110Z/ptcDataset/ptcDataset_94_LSSTCam_ptcTestResultado_07_20201231T024110Z.fits --detNum=94 --outDir=/project/plazas/WORK/DM-23159/TEST_BOT/ptcTestResultado.06/PDF_PLOTS_FULLCOVARIANCE PDF ( FULLCOVARIANCE with gen3 code): PTC_det94_FULLCOVARIANCE_DM-23159.pdf

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 07/Jan/21 2:17 AM - edited

A new test was added to ci_cpp_gen3 running basically the following command:

pipetask run -d "detector=0 and exposure IN (2020012800028, 2020012800029, 2020012800030, 2020012800031, 2020012800032, 2020012800033, 2020012800034, 2020012800035, 2020012800036, 2020012800037, 2020012800038, 2020012800039, 2020012800040, 2020012800041, 2020012800042, 2020012800043, 2020012800044, 2020012800045, 2020012800046, 2020012800047, 2020012800048, 2020012800049, 2020012800050, 2020012800051, 2020012800052, 2020012800053, 2020012800054, 2020012800055, 2020012800056, 2020012800057, 2020012800058, 2020012800059, 2020012800060, 2020012800061, 2020012800062, 2020012800063, 2020012800064, 2020012800065, 2020012800066, 2020012800067, 2020012800068, 2020012800069, 2020012800070, 2020012800071, 2020012800072, 2020012800073, 2020012800074, 2020012800075, 2020012800076, 2020012800077, 2020012800078, 2020012800079, 2020012800080, 2020012800081, 2020012800082, 2020012800083, 2020012800084, 2020012800085, 2020012800086, 2020012800087, 2020012800088, 2020012800089, 2020012800090, 2020012800091, 2020012800092, 2020012800093, 2020012800094, 2020012800095, 2020012800096, 2020012800097, 2020012800098, 2020012800099, 2020012800100, 2020012800101, 2020012800102, 2020012800103, 2020012800104, 2020012800105, 2020012800106, 2020012800107) and instrument = 'LATISS'" -b ./DATA/butler.yaml -i LATISS/raw/all,LATISS/calib,calib/v00 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION -o ptcTestResultado.204 --register-dataset-types

Show

Andrés Alejandro Plazas Malagón added a comment - 07/Jan/21 2:17 AM - edited A new test was added to ci_cpp_gen3 running basically the following command: pipetask run -d "detector=0 and exposure IN (2020012800028, 2020012800029, 2020012800030, 2020012800031, 2020012800032, 2020012800033, 2020012800034, 2020012800035, 2020012800036, 2020012800037, 2020012800038, 2020012800039, 2020012800040, 2020012800041, 2020012800042, 2020012800043, 2020012800044, 2020012800045, 2020012800046, 2020012800047, 2020012800048, 2020012800049, 2020012800050, 2020012800051, 2020012800052, 2020012800053, 2020012800054, 2020012800055, 2020012800056, 2020012800057, 2020012800058, 2020012800059, 2020012800060, 2020012800061, 2020012800062, 2020012800063, 2020012800064, 2020012800065, 2020012800066, 2020012800067, 2020012800068, 2020012800069, 2020012800070, 2020012800071, 2020012800072, 2020012800073, 2020012800074, 2020012800075, 2020012800076, 2020012800077, 2020012800078, 2020012800079, 2020012800080, 2020012800081, 2020012800082, 2020012800083, 2020012800084, 2020012800085, 2020012800086, 2020012800087, 2020012800088, 2020012800089, 2020012800090, 2020012800091, 2020012800092, 2020012800093, 2020012800094, 2020012800095, 2020012800096, 2020012800097, 2020012800098, 2020012800099, 2020012800100, 2020012800101, 2020012800102, 2020012800103, 2020012800104, 2020012800105, 2020012800106, 2020012800107) and instrument = 'LATISS'" -b ./DATA/butler.yaml -i LATISS/raw/all,LATISS/calib,calib/v00 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=EXPAPPROXIMATION -o ptcTestResultado.204 --register-dataset-types

Andrés Alejandro Plazas Malagón made changes - 07/Jan/21 2:46 AM

Attachment

ci_cpp_gen3_DM-23159.txt [ 47075 ]

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 07/Jan/21 2:46 AM

Scons has built successfully on ci_cpp_gen3: ci_cpp_gen3_DM-23159.txt

Show

Andrés Alejandro Plazas Malagón added a comment - 07/Jan/21 2:46 AM Scons has built successfully on ci_cpp_gen3 : ci_cpp_gen3_DM-23159.txt

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 07/Jan/21 8:44 PM

Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33335/pipeline

Show

Andrés Alejandro Plazas Malagón added a comment - 07/Jan/21 8:44 PM Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33335/pipeline

Andrés Alejandro Plazas Malagón made changes - 07/Jan/21 9:09 PM

Reviewers		Christopher Waters, Nate Lust [ cwaters, nlust ]
Status	In Progress [ 3 ]	In Review [ 10004 ]

Andrés Alejandro Plazas Malagón made changes - 07/Jan/21 9:51 PM

Story Points

3

Andrés Alejandro Plazas Malagón made changes - 07/Jan/21 9:53 PM

Story Points

23

Hide

Permalink

Nate Lust added a comment - 13/Jan/21 5:00 PM

Comments are up in cp_pipe

Show

Nate Lust added a comment - 13/Jan/21 5:00 PM Comments are up in cp_pipe

Hide

Permalink

Nate Lust added a comment - 21/Jan/21 6:18 PM

Thanks for the changes. Overall looks fine, I get why you would want to keep code similar to what you ported from for now. A few minor comments. I will ask that you file a ticket describing what work you want to do next to clean things up or reorganize and what the plan to do that will be, even if that does not occur until you get rid of gen2 support. This will help make sure nothing falls through the cracks.

Show

Nate Lust added a comment - 21/Jan/21 6:18 PM Thanks for the changes. Overall looks fine, I get why you would want to keep code similar to what you ported from for now. A few minor comments. I will ask that you file a ticket describing what work you want to do next to clean things up or reorganize and what the plan to do that will be, even if that does not occur until you get rid of gen2 support. This will help make sure nothing falls through the cracks.

Nate Lust made changes - 21/Jan/21 6:18 PM

Reviewers

Christopher Waters, Nate Lust [ cwaters, nlust ]

Christopher Waters [ cwaters ]

Hide

Permalink

Nate Lust added a comment - 21/Jan/21 6:19 PM

I took myself off reviewer list

Show

Nate Lust added a comment - 21/Jan/21 6:19 PM I took myself off reviewer list

Christopher Waters made changes - 21/Jan/21 9:57 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 22/Jan/21 1:12 AM

We had to run Jenkins with only ci_cpp_gen3 due to a ci_cpp_gen2 problem with the mac build: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33396/

But, I was able to successfully build ci_cpp_gen2: ci_cpp_gen2_output_2021JAN20.txt

Show

Andrés Alejandro Plazas Malagón added a comment - 22/Jan/21 1:12 AM We had to run Jenkins with only ci_cpp_gen3 due to a ci_cpp_gen2 problem with the mac build: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33396/ But, I was able to successfully build ci_cpp_gen2 : ci_cpp_gen2_output_2021JAN20.txt

Andrés Alejandro Plazas Malagón made changes - 22/Jan/21 1:12 AM

Attachment

ci_cpp_gen2_output_2021JAN20.txt [ 47368 ]

Andrés Alejandro Plazas Malagón made changes - 22/Jan/21 1:24 AM

Story Points

23

25

Andrés Alejandro Plazas Malagón made changes - 22/Jan/21 1:24 AM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

Hide

Permalink

Andrés Alejandro Plazas Malagón added a comment - 01/Feb/21 10:50 PM

Future PTC work: ~~DM-28588~~

Show

Andrés Alejandro Plazas Malagón added a comment - 01/Feb/21 10:50 PM Future PTC work: DM-28588

Data Management

Make PTC task compatible with RFC-352

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

CI Builds