`measurePothonTransferCurve.py` creates a 2D NumPy array for Linearize, and also produces `c0` coefficients per amplifier for `LinearizedSquared` (DM-21221). I persisted the table and the coefficients as yamls files by using

dataRef.put(lookupTableArray, datasetType="linearizerLut")
dataRef.put(dataset.coefficientLinearizeSquared, datasetType="linearizerSquared")

and adding

linearizerLut:
level: Ccd
persistable: lsst.ip.isr.linearize.LinearizeLookupTable
python: lsst.ip.isr.linearize.LinearizeLookupTable
storage: YamlStorage
tables:

• raw
  template: calibrations/linearizers/linearizer-lut-%(detectorName)s-%(detector)03d.yaml

linearizerSquared:
level: Ccd
persistable: lsst.ip.isr.linearize.LinearizeSquared
python: lsst.ip.isr.linearize.LinearizeSquared
storage: YamlStorage
tables:

• raw
  template: calibrations/linearizers/linearizer-squared-%(detectorName)s-%(detector)03d.yaml

in `obs_lsst/policy/lsstCamMapper.yaml`;

and also adding

linearizerLut:
description: "Lookup table linearizer from MeasurePhotonTransferCurveTask"
persistable: ignored
python: lsst.ip.isr.linearize.LinearizeLookuptable
storage: YamlStorage
tables: 'raw'
template: ''

linearizerSquared:
description: "Quadratic polynomial coefficients for linearizer from MeasurePhotonTransferCurveTask"
persistable: ignored
python: lsst.ip.isr.linearize.LinearizeSquared
storage: YamlStorage
tables: 'raw'
template: ''

in obs_base/policy/dataset.yaml.

The command run to produce the linearizers is as follows (w_2020_09):

measurePhotonTransferCurve.py /project/shared/auxTel/ --rerun plazas/test_lin_true --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 -c isr.doBias=False isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-versions --clobber-config

I then copied the yaml files to obs_lsst_data/latiss/SQ_linearizers/rx_s00/20200303T000000.yaml
and obs_lsst_data/latiss/LUT_linearizers/rx_s00/20200303T000000.yaml. (following the convention for the defects in /obs_lsst_data/latiss/defects/rxx_s00, and following the suggestion by Tim Jenness...although he did mention that this had to be done for gen3, but I am working in gen2).

Now that these files are persisted, I was hoping that maybe the linearity correction would be applied if I ran the same command as above but with isr.doLinearize=False. Alas, I got:

File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in _call_
result = self.runTask(task, dataRef, kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/home/plazas/lsst_devel/LSST/cp_pipe/python/lsst/cp/pipe/ptc.py", line 334, in runDataRef
exp1 = self.isr.runDataRef(dataRef).exposure
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 1552, in runDataRef
isrData = self.readIsrData(sensorRef, ccdExposure)
File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 961, in readIsrData
if self.doLinearize(ccd) else None)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1397, in get
location = self._locate(datasetType, dataId, write=False)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1316, in _locate
location = repoData.repo.map(datasetType, dataId, write=write)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/repository.py", line 239, in map
loc = self._mapper.map(*args, **kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/mapper.py", line 163, in map
return func(self.validate(dataId), write)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/cameraMapper.py", line 387, in mapClosure
return mapping.map(mapper, dataId, write)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 153, in map
actualId = self.need(iter(self.keyDict.keys()), dataId)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 317, in need
lookups = self.lookup(newProps, newId)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 262, in lookup
result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/registries.py", line 350, in lookup
cmd += " FROM " + " NATURAL JOIN ".join(reference)
TypeError: sequence item 0: expected str instance, NoneType found

Thanks for this. I've made quite extensive comments and would like some changes. How big are the lookup tables? It seems to me that the reason Linearizer can't be trivially written to a ecsv table is the optional lookup table. It might be quicker to read an ecsv table than parse a YAML document and in theory we could stuff the lookup table in the ecsv yaml header.

My main question about Linearizer itself is whether we really do envisage that different amps will have different linearity types. None of the code is doing per-amp changes of linearity type but the code supports it.

Thank you for your comments. I have replied to them and pushed the code again. It passed the Jenkins tests (John Swinbank Simon Krughoff): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31471/pipeline