`measurePothonTransferCurve.py` creates a 2D NumPy array for Linearize, and also produces `c0` coefficients per amplifier for `LinearizedSquared` (DM-21221). I persisted the table and the coefficients as yamls files by using
dataRef.put(lookupTableArray, datasetType="linearizerLut")
dataRef.put(dataset.coefficientLinearizeSquared, datasetType="linearizerSquared")
and adding
linearizerLut:
level: Ccd
persistable: lsst.ip.isr.linearize.LinearizeLookupTable
python: lsst.ip.isr.linearize.LinearizeLookupTable
storage: YamlStorage
tables:
- raw
template: calibrations/linearizers/linearizer-lut-%(detectorName)s-%(detector)03d.yaml
linearizerSquared:
level: Ccd
persistable: lsst.ip.isr.linearize.LinearizeSquared
python: lsst.ip.isr.linearize.LinearizeSquared
storage: YamlStorage
tables:
- raw
template: calibrations/linearizers/linearizer-squared-%(detectorName)s-%(detector)03d.yaml
in `obs_lsst/policy/lsstCamMapper.yaml`;
and also adding
linearizerLut:
description: "Lookup table linearizer from MeasurePhotonTransferCurveTask"
persistable: ignored
python: lsst.ip.isr.linearize.LinearizeLookuptable
storage: YamlStorage
tables: 'raw'
template: ''
linearizerSquared:
description: "Quadratic polynomial coefficients for linearizer from MeasurePhotonTransferCurveTask"
persistable: ignored
python: lsst.ip.isr.linearize.LinearizeSquared
storage: YamlStorage
tables: 'raw'
template: ''
in obs_base/policy/dataset.yaml.
The command run to produce the linearizers is as follows (w_2020_09):
measurePhotonTransferCurve.py /project/shared/auxTel/ --rerun plazas/test_lin_true --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 -c isr.doBias=False isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-versions --clobber-config
I then copied the yaml files to obs_lsst_data/latiss/SQ_linearizers/rx_s00/20200303T000000.yaml
and obs_lsst_data/latiss/LUT_linearizers/rx_s00/20200303T000000.yaml. (following the convention for the defects in /obs_lsst_data/latiss/defects/rxx_s00, and following the suggestion by Tim Jenness...although he did mention that this had to be done for gen3, but I am working in gen2).
Now that these files are persisted, I was hoping that maybe the linearity correction would be applied if I ran the same command as above but with isr.doLinearize=False. Alas, I got:
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in _call_
result = self.runTask(task, dataRef, kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/home/plazas/lsst_devel/LSST/cp_pipe/python/lsst/cp/pipe/ptc.py", line 334, in runDataRef
exp1 = self.isr.runDataRef(dataRef).exposure
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 1552, in runDataRef
isrData = self.readIsrData(sensorRef, ccdExposure)
File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 961, in readIsrData
if self.doLinearize(ccd) else None)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1397, in get
location = self._locate(datasetType, dataId, write=False)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1316, in _locate
location = repoData.repo.map(datasetType, dataId, write=write)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/repository.py", line 239, in map
loc = self._mapper.map(*args, **kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/mapper.py", line 163, in map
return func(self.validate(dataId), write)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/cameraMapper.py", line 387, in mapClosure
return mapping.map(mapper, dataId, write)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 153, in map
actualId = self.need(iter(self.keyDict.keys()), dataId)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 317, in need
lookups = self.lookup(newProps, newId)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 262, in lookup
result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/registries.py", line 350, in lookup
cmd += " FROM " + " NATURAL JOIN ".join(reference)
TypeError: sequence item 0: expected str instance, NoneType found
`measurePothonTransferCurve.py` creates a 2D NumPy array for Linearize, and also produces `c0` coefficients per amplifier for `LinearizedSquared` (
DM-21221). I persisted the table and the coefficients as yamls files by usingdataRef.put(lookupTableArray, datasetType="linearizerLut")
dataRef.put(dataset.coefficientLinearizeSquared, datasetType="linearizerSquared")
and adding
linearizerLut:
level: Ccd
persistable: lsst.ip.isr.linearize.LinearizeLookupTable
python: lsst.ip.isr.linearize.LinearizeLookupTable
storage: YamlStorage
tables:
template: calibrations/linearizers/linearizer-lut-%(detectorName)s-%(detector)03d.yaml
linearizerSquared:
level: Ccd
persistable: lsst.ip.isr.linearize.LinearizeSquared
python: lsst.ip.isr.linearize.LinearizeSquared
storage: YamlStorage
tables:
template: calibrations/linearizers/linearizer-squared-%(detectorName)s-%(detector)03d.yaml
in `obs_lsst/policy/lsstCamMapper.yaml`;
and also adding
linearizerLut:
description: "Lookup table linearizer from MeasurePhotonTransferCurveTask"
persistable: ignored
python: lsst.ip.isr.linearize.LinearizeLookuptable
storage: YamlStorage
tables: 'raw'
template: ''
linearizerSquared:
description: "Quadratic polynomial coefficients for linearizer from MeasurePhotonTransferCurveTask"
persistable: ignored
python: lsst.ip.isr.linearize.LinearizeSquared
storage: YamlStorage
tables: 'raw'
template: ''
in obs_base/policy/dataset.yaml.
The command run to produce the linearizers is as follows (w_2020_09):
measurePhotonTransferCurve.py /project/shared/auxTel/ --rerun plazas/test_lin_true --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 -c isr.doBias=False isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-versions --clobber-config
I then copied the yaml files to obs_lsst_data/latiss/SQ_linearizers/rx_s00/20200303T000000.yaml
and obs_lsst_data/latiss/LUT_linearizers/rx_s00/20200303T000000.yaml. (following the convention for the defects in /obs_lsst_data/latiss/defects/rxx_s00, and following the suggestion by Tim Jenness...although he did mention that this had to be done for gen3, but I am working in gen2).
Now that these files are persisted, I was hoping that maybe the linearity correction would be applied if I ran the same command as above but with isr.doLinearize=False. Alas, I got:
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in _call_
result = self.runTask(task, dataRef, kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/home/plazas/lsst_devel/LSST/cp_pipe/python/lsst/cp/pipe/ptc.py", line 334, in runDataRef
exp1 = self.isr.runDataRef(dataRef).exposure
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
res = func(self, *args, **keyArgs)
File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 1552, in runDataRef
isrData = self.readIsrData(sensorRef, ccdExposure)
File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 961, in readIsrData
if self.doLinearize(ccd) else None)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1397, in get
location = self._locate(datasetType, dataId, write=False)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1316, in _locate
location = repoData.repo.map(datasetType, dataId, write=write)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/repository.py", line 239, in map
loc = self._mapper.map(*args, **kwargs)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/mapper.py", line 163, in map
return func(self.validate(dataId), write)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/cameraMapper.py", line 387, in mapClosure
return mapping.map(mapper, dataId, write)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 153, in map
actualId = self.need(iter(self.keyDict.keys()), dataId)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 317, in need
lookups = self.lookup(newProps, newId)
File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 262, in lookup
result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/registries.py", line 350, in lookup
cmd += " FROM " + " NATURAL JOIN ".join(reference)
TypeError: sequence item 0: expected str instance, NoneType found