Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18053

Build linearity models for the AuxTel sensor

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Deliverable is a linearity model for each amplifier in the AuxTel sensor for its current configuration. Having some code to easily recalculate it is important, as this will surely change when they start switching/shielding cables etc

      This will almost certainly involve getting the LATISS lab-folk to acquire and transfer the necessary data, and will also probably involve talking to Pierre^2 about how to take the type of nice ramp-like frames they presented at the PCWG meeting in Paris.

      This will probably be interesting work as this is the first time we will have looked at this type of data.

      It may also be hairer than expected, as although there is code in the stack for applying linearity correction, I don't think it has been exercised much, if at all. (A good start would be checking if every single obs_package has isr.doLinearize=False in its ISR config, and if that is the case, expect complications.)

        Attachments

          Activity

          Hide
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited

          As part of DM-23045, auxTel data was used to produced polynomial linearizers by running the Photon Transfer Curve task.

          Flat pairs were taken on 2020-02-21: https://confluence.lsstcorp.org/display/LSSTCOM/2020-02-21 , IDs: 13-36

          Data ID's (pairs of flats at different exposure times):

          2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036
          

          Example command to create the linearizer (isr.doLinearize=False):

          measurePhotonTransferCurve.py /project/plazas/auxTel-local-copy-2020MAR30/ --rerun plazas/test_2020APR04-lin-false_POLY_DEG_3 --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 instrumentName=LATISS polynomialFitDegreeNonLinearity=3 -c isr.doBias=True isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-config --clobber-versions

          The following parameter controls the order of the polynomial fit (fir of the meanSignal vs expTime curve):

          polynomialFitDegreeNonLinearity=3
          

          The above command produces fits files that contain a quadratic linearizer, a polynomial linearizer, and a LUT table formed with the linear part of the polynomial. They can be found in:

          /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_2/calibrations/linearizers/
           
          /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_3/calibrations/linearizers/
           
          /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_4/calibrations/linearizers/
          

          Files for the polynomial linearizers:
          Polynomial degree 2: linearize-polynomial-DEG2_det000_2020-04-14.fits
          Polynomial degree 3: linearize-polynomial-DEG3_det000_2020-04-13.fits
          Polynomial degree 4: linearize-polynomial-DEG4_det000_2020-04-14.fits

          To use the linearizers, they need to be ingested in the CALIB registry of the repo that you are using (in this case, /project/plazas/auxTel-local-copy-2020MAR30/CALIB). Before that, the fits file needs to be converted to text file (e.g., yaml) and the filename must be changed to something of the form "YYYY-MM-DDThh:mm:ss.yaml". The name in this file has to match the information on the metadata of the fits file if they are different ingestCuratedCalibs.py will tell you (Edit: probably not anymore, see comment below by Tim):

          ValueError: Path and file metadata do not agree:
          Path metadata: LATISS 0 2020-03-30T00:00:00 linearizer
          File metadata: LATISS 0 2020-03-30T18:12:25.459517 linearizer
          File read from : /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/rxx_s00/2020-03-30T00:00:00.yaml
          

          To convert the fits fits to yaml, you can use: convertFitsToYaml.py, and then change the name of the resulting file to something like:

          2020:04:14T00:00:00.yaml
          

          To ingest the files in the calibration registry, use:

          ingestCuratedCalibs.py /project/plazas/auxTel-local-copy-2020MAR30/ /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/ --calib /project/plazas/auxTel-local-copy-2020MAR30/CALIB
          

          Note that the above command will ingest all the ecvs and yaml files that it finds in /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/. It is also important that the files are in "/home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/rxx_s00" (the last three directories: "INSTRUMENT/linearizer/DETECTOR"). "ingestCuratedCalibs.py" is relying on this directory structure, otherwise it will complain.

          Finally, you may need to go to the sql calib registry and change the validityStart date of the linearizer, (unless "your data that are being processed [are] taken after today"). Something like:

          sqlite3 /project/plazas/auxTel-local-copy-2020MAR30/CALIB/calibRegistry.sqlite3
          sqlite> UPDATE linearizer SET validStart='2020-01-01' WHERE id = 1;
          

          Now you should be able to do

           doLinearize=True 

          in the above PTC task command to apply the linearizer. Fractional NL residual from applying the linearizers attached to this post, to the same auxTel data described here (taken from DM-23045):

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited As part of DM-23045 , auxTel data was used to produced polynomial linearizers by running the Photon Transfer Curve task. Flat pairs were taken on 2020-02-21: https://confluence.lsstcorp.org/display/LSSTCOM/2020-02-21 , IDs: 13-36 Data ID's (pairs of flats at different exposure times): 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 Example command to create the linearizer (isr.doLinearize=False): measurePhotonTransferCurve.py /project/plazas/auxTel-local-copy-2020MAR30/ --rerun plazas/test_2020APR04-lin-false_POLY_DEG_3 --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 instrumentName=LATISS polynomialFitDegreeNonLinearity=3 -c isr.doBias=True isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-config --clobber-versions The following parameter controls the order of the polynomial fit (fir of the meanSignal vs expTime curve): polynomialFitDegreeNonLinearity=3 The above command produces fits files that contain a quadratic linearizer, a polynomial linearizer, and a LUT table formed with the linear part of the polynomial. They can be found in: /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_2/calibrations/linearizers/   /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_3/calibrations/linearizers/   /project/plazas/auxTel-local-copy-2020MAR30/rerun/plazas/test_2020APR13-lin-false_POLY_DEG_4/calibrations/linearizers/ Files for the polynomial linearizers: Polynomial degree 2: linearize-polynomial-DEG2_det000_2020-04-14.fits Polynomial degree 3: linearize-polynomial-DEG3_det000_2020-04-13.fits Polynomial degree 4: linearize-polynomial-DEG4_det000_2020-04-14.fits To use the linearizers, they need to be ingested in the CALIB registry of the repo that you are using (in this case, /project/plazas/auxTel-local-copy-2020MAR30/CALIB). Before that, the fits file needs to be converted to text file (e.g., yaml) and the filename must be changed to something of the form "YYYY-MM-DDThh:mm:ss.yaml". The name in this file has to match the information on the metadata of the fits file if they are different ingestCuratedCalibs.py will tell you (Edit: probably not anymore, see comment below by Tim): ValueError: Path and file metadata do not agree: Path metadata: LATISS 0 2020-03-30T00:00:00 linearizer File metadata: LATISS 0 2020-03-30T18:12:25.459517 linearizer File read from : /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/rxx_s00/2020-03-30T00:00:00.yaml To convert the fits fits to yaml, you can use: convertFitsToYaml.py , and then change the name of the resulting file to something like: 2020:04:14T00:00:00.yaml To ingest the files in the calibration registry, use: ingestCuratedCalibs.py /project/plazas/auxTel-local-copy-2020MAR30/ /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/ --calib /project/plazas/auxTel-local-copy-2020MAR30/CALIB Note that the above command will ingest all the ecvs and yaml files that it finds in /home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/. It is also important that the files are in "/home/plazas/lsst_devel/aDM-23045/LATISS/linearizer/rxx_s00" (the last three directories: "INSTRUMENT/linearizer/DETECTOR"). "ingestCuratedCalibs.py" is relying on this directory structure, otherwise it will complain. Finally, you may need to go to the sql calib registry and change the validityStart date of the linearizer, (unless "your data that are being processed [are] taken after today"). Something like: sqlite3 /project/plazas/auxTel-local-copy-2020MAR30/CALIB/calibRegistry.sqlite3 sqlite> UPDATE linearizer SET validStart='2020-01-01' WHERE id = 1; Now you should be able to do doLinearize=True in the above PTC task command to apply the linearizer. Fractional NL residual from applying the linearizers attached to this post, to the same auxTel data described here (taken from DM-23045 ):
          Hide
          tjenness Tim Jenness added a comment -

          I think the calibDate problem you are reporting was fixed in DM-24027.

          Show
          tjenness Tim Jenness added a comment - I think the calibDate problem you are reporting was fixed in DM-24027 .
          Hide
          plazas Andrés Alejandro Plazas Malagón added a comment -

          I had not checked that, but that is good to know. Thank you for the heads-up.

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - I had not checked that, but that is good to know. Thank you for the heads-up.

            People

            • Assignee:
              plazas Andrés Alejandro Plazas Malagón
              Reporter:
              mfisherlevine Merlin Fisher-Levine
              Watchers:
              Andrés Alejandro Plazas Malagón, John Swinbank, Merlin Fisher-Levine, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel