Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23044

PTC task should persist usable linearity models

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe, obs_base
    • Labels:
      None

      Description

      Store the output of DM-21221 in some meaningful form using the Butler, and demonstrate that it can be used in downstream processing.

        Attachments

          Issue Links

            Activity

            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            `measurePothonTransferCurve.py` creates a 2D NumPy array for Linearize, and also produces `c0` coefficients per amplifier for `LinearizedSquared` (DM-21221). I persisted the table and the coefficients as yamls files by using

            dataRef.put(lookupTableArray, datasetType="linearizerLut")
            dataRef.put(dataset.coefficientLinearizeSquared, datasetType="linearizerSquared")

            and adding

            linearizerLut:
            level: Ccd
            persistable: lsst.ip.isr.linearize.LinearizeLookupTable
            python: lsst.ip.isr.linearize.LinearizeLookupTable
            storage: YamlStorage
            tables:

            • raw
              template: calibrations/linearizers/linearizer-lut-%(detectorName)s-%(detector)03d.yaml

            linearizerSquared:
            level: Ccd
            persistable: lsst.ip.isr.linearize.LinearizeSquared
            python: lsst.ip.isr.linearize.LinearizeSquared
            storage: YamlStorage
            tables:

            • raw
              template: calibrations/linearizers/linearizer-squared-%(detectorName)s-%(detector)03d.yaml

            in `obs_lsst/policy/lsstCamMapper.yaml`;

            and also adding

            linearizerLut:
            description: "Lookup table linearizer from MeasurePhotonTransferCurveTask"
            persistable: ignored
            python: lsst.ip.isr.linearize.LinearizeLookuptable
            storage: YamlStorage
            tables: 'raw'
            template: ''

            linearizerSquared:
            description: "Quadratic polynomial coefficients for linearizer from MeasurePhotonTransferCurveTask"
            persistable: ignored
            python: lsst.ip.isr.linearize.LinearizeSquared
            storage: YamlStorage
            tables: 'raw'
            template: ''

            in obs_base/policy/dataset.yaml.

            The command run to produce the linearizers is as follows (w_2020_09):

            measurePhotonTransferCurve.py /project/shared/auxTel/ --rerun plazas/test_lin_true --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 -c isr.doBias=False isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-versions --clobber-config

            I then copied the yaml files to obs_lsst_data/latiss/SQ_linearizers/rx_s00/20200303T000000.yaml
            and obs_lsst_data/latiss/LUT_linearizers/rx_s00/20200303T000000.yaml. (following the convention for the defects in /obs_lsst_data/latiss/defects/rxx_s00, and following the suggestion by Tim Jenness...although he did mention that this had to be done for gen3, but I am working in gen2).

            Now that these files are persisted, I was hoping that maybe the linearity correction would be applied if I ran the same command as above but with isr.doLinearize=False. Alas, I got:

            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in _call_
            result = self.runTask(task, dataRef, kwargs)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
            return task.runDataRef(dataRef, **kwargs)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
            res = func(self, *args, **keyArgs)
            File "/home/plazas/lsst_devel/LSST/cp_pipe/python/lsst/cp/pipe/ptc.py", line 334, in runDataRef
            exp1 = self.isr.runDataRef(dataRef).exposure
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper
            res = func(self, *args, **keyArgs)
            File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 1552, in runDataRef
            isrData = self.readIsrData(sensorRef, ccdExposure)
            File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 961, in readIsrData
            if self.doLinearize(ccd) else None)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
            return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1397, in get
            location = self._locate(datasetType, dataId, write=False)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1316, in _locate
            location = repoData.repo.map(datasetType, dataId, write=write)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/repository.py", line 239, in map
            loc = self._mapper.map(*args, **kwargs)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/mapper.py", line 163, in map
            return func(self.validate(dataId), write)
            File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/cameraMapper.py", line 387, in mapClosure
            return mapping.map(mapper, dataId, write)
            File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 153, in map
            actualId = self.need(iter(self.keyDict.keys()), dataId)
            File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 317, in need
            lookups = self.lookup(newProps, newId)
            File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 262, in lookup
            result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
            File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/registries.py", line 350, in lookup
            cmd += " FROM " + " NATURAL JOIN ".join(reference)
            TypeError: sequence item 0: expected str instance, NoneType found

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - `measurePothonTransferCurve.py` creates a 2D NumPy array for Linearize, and also produces `c0` coefficients per amplifier for `LinearizedSquared` ( DM-21221 ). I persisted the table and the coefficients as yamls files by using dataRef.put(lookupTableArray, datasetType="linearizerLut") dataRef.put(dataset.coefficientLinearizeSquared, datasetType="linearizerSquared") and adding linearizerLut: level: Ccd persistable: lsst.ip.isr.linearize.LinearizeLookupTable python: lsst.ip.isr.linearize.LinearizeLookupTable storage: YamlStorage tables: raw template: calibrations/linearizers/linearizer-lut-%(detectorName)s-%(detector)03d.yaml linearizerSquared: level: Ccd persistable: lsst.ip.isr.linearize.LinearizeSquared python: lsst.ip.isr.linearize.LinearizeSquared storage: YamlStorage tables: raw template: calibrations/linearizers/linearizer-squared-%(detectorName)s-%(detector)03d.yaml in `obs_lsst/policy/lsstCamMapper.yaml`; and also adding linearizerLut: description: "Lookup table linearizer from MeasurePhotonTransferCurveTask" persistable: ignored python: lsst.ip.isr.linearize.LinearizeLookuptable storage: YamlStorage tables: 'raw' template: '' linearizerSquared: description: "Quadratic polynomial coefficients for linearizer from MeasurePhotonTransferCurveTask" persistable: ignored python: lsst.ip.isr.linearize.LinearizeSquared storage: YamlStorage tables: 'raw' template: '' in obs_base/policy/dataset.yaml. The command run to produce the linearizers is as follows (w_2020_09): measurePhotonTransferCurve.py /project/shared/auxTel/ --rerun plazas/test_lin_true --id detector=0 --visit-pairs 2020022100013,2020022100014 2020022100015,2020022100016 2020022100017,2020022100018 2020022100019,2020022100020 2020022100021,2020022100022 2020022100023,2020022100024 2020022100025,2020022100026 2020022100027,2020022100028 2020022100029,2020022100030 2020022100031,2020022100032 2020022100033,2020022100034 2020022100035,2020022100036 -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 -c isr.doBias=False isr.doDark=False isr.doSuspect=False isr.doLinearize=False --clobber-versions --clobber-config I then copied the yaml files to obs_lsst_data/latiss/SQ_linearizers/rx_s00/20200303T000000.yaml and obs_lsst_data/latiss/LUT_linearizers/rx_s00/20200303T000000.yaml. (following the convention for the defects in /obs_lsst_data/latiss/defects/rxx_s00, and following the suggestion by Tim Jenness ...although he did mention that this had to be done for gen3, but I am working in gen2). Now that these files are persisted, I was hoping that maybe the linearity correction would be applied if I ran the same command as above but with isr.doLinearize=False. Alas, I got: File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 388, in _ call _ result = self.runTask(task, dataRef, kwargs) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask return task.runDataRef(dataRef, **kwargs) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/home/plazas/lsst_devel/LSST/cp_pipe/python/lsst/cp/pipe/ptc.py", line 334, in runDataRef exp1 = self.isr.runDataRef(dataRef).exposure File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/pipe_base/19.0.0-9-g0ae078d+4/python/lsst/pipe/base/timer.py", line 150, in wrapper res = func(self, *args, **keyArgs) File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 1552, in runDataRef isrData = self.readIsrData(sensorRef, ccdExposure) File "/home/plazas/lsst_devel/LSST/ip_isr/python/lsst/ip/isr/isrTask.py", line 961, in readIsrData if self.doLinearize(ccd) else None) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butlerSubset.py", line 203, in get return self.butlerSubset.butler.get(datasetType, self.dataId, **rest) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1397, in get location = self._locate(datasetType, dataId, write=False) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/butler.py", line 1316, in _locate location = repoData.repo.map(datasetType, dataId, write=write) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/repository.py", line 239, in map loc = self._mapper.map(*args, **kwargs) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/mapper.py", line 163, in map return func(self.validate(dataId), write) File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/cameraMapper.py", line 387, in mapClosure return mapping.map(mapper, dataId, write) File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 153, in map actualId = self.need(iter(self.keyDict.keys()), dataId) File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 317, in need lookups = self.lookup(newProps, newId) File "/home/plazas/lsst_devel/LSST/obs_base/python/lsst/obs/base/mapping.py", line 262, in lookup result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template) File "/software/lsstsw/stack_20200220/stack/miniconda3-4.7.12-984c9f7/Linux64/daf_persistence/19.0.0-2-gf01c5b1/python/lsst/daf/persistence/registries.py", line 350, in lookup cmd += " FROM " + " NATURAL JOIN ".join(reference) TypeError: sequence item 0: expected str instance, NoneType found
            Hide
            tjenness Tim Jenness added a comment -

            Thanks for this. I've made quite extensive comments and would like some changes. How big are the lookup tables? It seems to me that the reason Linearizer can't be trivially written to a ecsv table is the optional lookup table. It might be quicker to read an ecsv table than parse a YAML document and in theory we could stuff the lookup table in the ecsv yaml header.

            My main question about Linearizer itself is whether we really do envisage that different amps will have different linearity types. None of the code is doing per-amp changes of linearity type but the code supports it.

            Show
            tjenness Tim Jenness added a comment - Thanks for this. I've made quite extensive comments and would like some changes. How big are the lookup tables? It seems to me that the reason Linearizer can't be trivially written to a ecsv table is the optional lookup table. It might be quicker to read an ecsv table than parse a YAML document and in theory we could stuff the lookup table in the ecsv yaml header. My main question about Linearizer itself is whether we really do envisage that different amps will have different linearity types. None of the code is doing per-amp changes of linearity type but the code supports it.
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Thank you for your comments. I have replied to them and pushed the code again. It passed the Jenkins tests (John Swinbank Simon Krughoff): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31471/pipeline

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Thank you for your comments. I have replied to them and pushed the code again. It passed the Jenkins tests ( John Swinbank Simon Krughoff ): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31471/pipeline

              People

              • Assignee:
                plazas Andrés Alejandro Plazas Malagón
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Tim Jenness
                Watchers:
                Andrés Alejandro Plazas Malagón, Christopher Waters, John Swinbank, Merlin Fisher-Levine, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel