# Build linearity models for the AuxTel sensor

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
8
• Team:
Data Release Production

## Description

Deliverable is a linearity model for each amplifier in the AuxTel sensor for its current configuration. Having some code to easily recalculate it is important, as this will surely change when they start switching/shielding cables etc

This will almost certainly involve getting the LATISS lab-folk to acquire and transfer the necessary data, and will also probably involve talking to Pierre^2 about how to take the type of nice ramp-like frames they presented at the PCWG meeting in Paris.

This will probably be interesting work as this is the first time we will have looked at this type of data.

It may also be hairer than expected, as although there is code in the stack for applying linearity correction, I don't think it has been exercised much, if at all. (A good start would be checking if every single obs_package has isr.doLinearize=False in its ISR config, and if that is the case, expect complications.)

## Activity

Hide
Andrés Alejandro Plazas Malagón added a comment - - edited

As part of DM-23045, auxTel data was used to produced polynomial linearizers by running the Photon Transfer Curve task.

Flat pairs were taken on 2020-02-21: https://confluence.lsstcorp.org/display/LSSTCOM/2020-02-21 , IDs: 13-36

Data ID's (pairs of flats at different exposure times):

 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 

Example command to create the linearizer (isr.doLinearize=False):

measurePhotonTransferCurve.py /project/plazas/auxTel-local-copy-2020MAR30/ --rerun plazas/test_2020APR04-lin-false_POLY_DEG_3 --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 instrumentName=LATISS polynomialFitDegreeNonLinearity=3 -c isr.doBias=True isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-config --clobber-versions

The following parameter controls the order of the polynomial fit (fir of the meanSignal vs expTime curve):

 polynomialFitDegreeNonLinearity=3 

The above command produces fits files that contain a quadratic linearizer, a polynomial linearizer, and a LUT table formed with the linear part of the polynomial. They can be found in:

 /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_2/calibrations/linearizers/   /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_3/calibrations/linearizers/   /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_4/calibrations/linearizers/ 

Files for the polynomial linearizers:
Polynomial degree 2: linearize-polynomial-DEG2_det000_2020-04-14.fits
Polynomial degree 3: linearize-polynomial-DEG3_det000_2020-04-13.fits
Polynomial degree 4: linearize-polynomial-DEG4_det000_2020-04-14.fits

To use the linearizers, they need to be ingested in the CALIB registry of the repo that you are using (in this case, /project/plazas/auxTel-local-copy-2020MAR30/CALIB). Before that, the fits file needs to be converted to text file (e.g., yaml) and the filename must be changed to something of the form "YYYY-MM-DDThh:mm:ss.yaml". The name in this file has to match the information on the metadata of the fits file if they are different ingestCuratedCalibs.py will tell you (Edit: probably not anymore, see comment below by Tim):

 ValueError: Path and file metadata do not agree: Path metadata: LATISS 0 2020-03-30T00:00:00 linearizer File metadata: LATISS 0 2020-03-30T18:12:25.459517 linearizer File read from : /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/rxx_s00/2020-03-30T00:00:00.yaml 

To convert the fits fits to yaml, you can use: convertFitsToYaml.py, and then change the name of the resulting file to something like:

 2020:04:14T00:00:00.yaml 

To ingest the files in the calibration registry, use:

 ingestCuratedCalibs.py /project/plazas/auxTel-local-copy-2020MAR30/ /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/ --calib /project/plazas/auxTel-local-copy-2020MAR30/CALIB 

Note that the above command will ingest all the ecvs and yaml files that it finds in /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/. It is also important that the files are in "/home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/rxx_s00" (the last three directories: "INSTRUMENT/linearizer/DETECTOR"). "ingestCuratedCalibs.py" is relying on this directory structure, otherwise it will complain.

Finally, you may need to go to the sql calib registry and change the validityStart date of the linearizer, (unless "your data that are being processed [are] taken after today"). Something like:

 sqlite3 /project/plazas/auxTel-local-copy-2020MAR30/CALIB/calibRegistry.sqlite3 sqlite> UPDATE linearizer SET validStart='2020-01-01' WHERE id = 1; 

Now you should be able to do

  doLinearize=True

in the above PTC task command to apply the linearizer. Fractional NL residual from applying the linearizers attached to this post, to the same auxTel data described here (taken from DM-23045):

Show
Hide
Tim Jenness added a comment -

I think the calibDate problem you are reporting was fixed in DM-24027.

Show
Tim Jenness added a comment - I think the calibDate problem you are reporting was fixed in DM-24027 .
Hide
Andrés Alejandro Plazas Malagón added a comment -

I had not checked that, but that is good to know. Thank you for the heads-up.

Show
Andrés Alejandro Plazas Malagón added a comment - I had not checked that, but that is good to know. Thank you for the heads-up.

## People

• Assignee:
Andrés Alejandro Plazas Malagón
Reporter:
Merlin Fisher-Levine
Watchers:
Andrés Alejandro Plazas Malagón, John Swinbank, Merlin Fisher-Levine, Tim Jenness