Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29412

Update StampsBase so that it can serve as StorageClass for subclasses

    XMLWordPrintable

    Details

      Description

      For ts_wep we want to create a new subclass based upon the StampsBase class in meas_algorithms. To do this we are going to change StampsBase `readFits` and `writeFits` methods to allow this according to the prescription Tim Jenness described on slack in [this thread|https://lsstc.slack.com/archives/C01FBUGM2CV/p1616625257255300.] We will then have to add StampsBase to the StorageClass.yaml file in daf_butler.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Specifically:

            1. In writeFits add a new primary header key "STAMPCLS" (8 characters is simplest) with the full name of the class being serialized.
            2. This name could come from lsst.daf.butler.core.utils.getFullTypeName (which means it probably should be moved to lsst.utils at some point)
            3. In readFits read the primary header, look for STAMPCLS, if it's there call lsst.utils.doImport on the class string.
            4. Use the return value from doImport to override the `cls` variable used for the class method.
            Show
            tjenness Tim Jenness added a comment - Specifically: In writeFits add a new primary header key "STAMPCLS" (8 characters is simplest) with the full name of the class being serialized. This name could come from lsst.daf.butler.core.utils.getFullTypeName (which means it probably should be moved to lsst.utils at some point) In readFits read the primary header, look for STAMPCLS, if it's there call lsst.utils.doImport on the class string. Use the return value from doImport to override the `cls` variable used for the class method.
            Hide
            tjenness Tim Jenness added a comment -

            Not sure you want to bypass the reading of the header if your cls variable is already a subclass of StampsBase (implying that someone is explicitly asking for a class and not using the base class to do it).

            Show
            tjenness Tim Jenness added a comment - Not sure you want to bypass the reading of the header if your cls variable is already a subclass of StampsBase (implying that someone is explicitly asking for a class and not using the base class to do it).
            Hide
            jbkalmbach Bryce Kalmbach added a comment - - edited

            Still seeing this error:

             

            `KeyError: "Unable to find item in registry with keys: donutStamps ({'instrument': 'LSSTCam-PhoSim'}), band+instrument+detector+physical_filter+exposure ({'instrument': 'LSSTCam-PhoSim'}), StampsBase ({'instrument': 'LSSTCam-PhoSim'}), donutStamps, band+instrument+detector+physical_filter+exposure, StampsBase"`

             

            Is there a step we are missing?

            Show
            jbkalmbach Bryce Kalmbach added a comment - - edited Still seeing this error:   `KeyError: "Unable to find item in registry with keys: donutStamps ({'instrument': 'LSSTCam-PhoSim'}), band+instrument+detector+physical_filter+exposure ({'instrument': 'LSSTCam-PhoSim'}), StampsBase ({'instrument': 'LSSTCam-PhoSim'}), donutStamps, band+instrument+detector+physical_filter+exposure, StampsBase"`   Is there a step we are missing?
            Hide
            jbkalmbach Bryce Kalmbach added a comment -
            Show
            jbkalmbach Bryce Kalmbach added a comment - My changes are pushed to github here:  https://github.com/lsst/meas_algorithms/compare/tickets/DM-29412
            Hide
            tjenness Tim Jenness added a comment -

            I can take a look at that meas_algorithms change if you make a pull request.

            Have you got more context for that KeyError you are getting?

            Show
            tjenness Tim Jenness added a comment - I can take a look at that meas_algorithms change if you make a pull request. Have you got more context for that KeyError you are getting?
            Hide
            jbkalmbach Bryce Kalmbach added a comment -

            Yeah, I have a full traceback here:

             

            Process task-5:
            Traceback (most recent call last):
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 659, in _determine_put_formatter_location
            formatter = self.formatterFactory.getFormatter(ref,
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 751, in getFormatter
            _, formatter = self.getFormatterWithMatch(entity, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 724, in getFormatterWithMatch
            matchKey, formatter = self._mappingFactory.getFromRegistryWithMatch(names, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 188, in getFromRegistryWithMatch
            key, cls, registry_kwargs = self.getClassFromRegistryWithMatch(targetClasses)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 134, in getClassFromRegistryWithMatch
            raise KeyError(f"Unable to find item in registry with key{plural}: {msg}")
            KeyError: "Unable to find item in registry with keys: donutStamps ({'instrument': 'LSSTCam-PhoSim'}), band+instrument+detector+physical_filter+exposure ({'instrument': 'LSSTCam-PhoSim'}), StampsBase ({'instrument': 'LSSTCam-PhoSim'}), donutStamps, band+instrument+detector+physical_filter+exposure, StampsBase"

            The above exception was the direct cause of the following exception:

            Traceback (most recent call last):
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
            self.run()
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 108, in run
            self._target(*self._args, **self._kwargs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 127, in _executeJob
            quantumExecutor.execute(taskDef, quantum, butler)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute
            self.runQuantum(task, quantum, taskDef, butler)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 274, in runQuantum
            task.runQuantum(butlerQC, inputRefs, outputRefs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/pipelineTask.py", line 156, in runQuantum
            butlerQC.put(outputs, outputRefs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 198, in put
            self._put(valuesAttribute, refs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 98, in _put
            butler.put(value, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner
            return func(self, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/_butler.py", line 851, in put
            self.datastore.put(obj, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner
            return func(self, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 1461, in put
            storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 952, in _write_in_memory_to_artifact
            location, formatter = self._prepare_for_put(inMemoryDataset, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 627, in _prepare_for_put
            return self._determine_put_formatter_location(ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 664, in _determine_put_formatter_location
            raise DatasetTypeNotSupportedError(f"Unable to find formatter for {ref} in datastore "
            lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError: Unable to find formatter for donutStamps@{instrument: 'LSSTCam-PhoSim', detector: 90, exposure: 9006001, ...}, sc=StampsBase] (id=21) in datastore FileDatastore@<butlerRoot>

             

            In the task I wrote the connections class has this to setup the donut stamps:

             

            donutStamps = connectionTypes.Output(
            doc="Donut Postage Stamp Images",
            dimensions=("exposure", "detector", "instrument"),
            storageClass="StampsBase",
            name="donutStamps",
            )

            Show
            jbkalmbach Bryce Kalmbach added a comment - Yeah, I have a full traceback here:   Process task-5: Traceback (most recent call last): File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 659, in _determine_put_formatter_location formatter = self.formatterFactory.getFormatter(ref, File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 751, in getFormatter _, formatter = self.getFormatterWithMatch(entity, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 724, in getFormatterWithMatch matchKey, formatter = self._mappingFactory.getFromRegistryWithMatch(names, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 188, in getFromRegistryWithMatch key, cls, registry_kwargs = self.getClassFromRegistryWithMatch(targetClasses) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 134, in getClassFromRegistryWithMatch raise KeyError(f"Unable to find item in registry with key{plural}: {msg}") KeyError: "Unable to find item in registry with keys: donutStamps ({'instrument': 'LSSTCam-PhoSim'}), band+instrument+detector+physical_filter+exposure ({'instrument': 'LSSTCam-PhoSim'}), StampsBase ({'instrument': 'LSSTCam-PhoSim'}), donutStamps, band+instrument+detector+physical_filter+exposure, StampsBase" The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 127, in _executeJob quantumExecutor.execute(taskDef, quantum, butler) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute self.runQuantum(task, quantum, taskDef, butler) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 274, in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/pipelineTask.py", line 156, in runQuantum butlerQC.put(outputs, outputRefs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 198, in put self._put(valuesAttribute, refs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 98, in _put butler.put(value, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner return func(self, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/_butler.py", line 851, in put self.datastore.put(obj, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner return func(self, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 1461, in put storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 952, in _write_in_memory_to_artifact location, formatter = self._prepare_for_put(inMemoryDataset, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 627, in _prepare_for_put return self._determine_put_formatter_location(ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 664, in _determine_put_formatter_location raise DatasetTypeNotSupportedError(f"Unable to find formatter for {ref} in datastore " lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError: Unable to find formatter for donutStamps@{instrument: 'LSSTCam-PhoSim', detector: 90, exposure: 9006001, ...}, sc=StampsBase] (id=21) in datastore FileDatastore@<butlerRoot>   In the task I wrote the connections class has this to setup the donut stamps:   donutStamps = connectionTypes.Output( doc="Donut Postage Stamp Images", dimensions=("exposure", "detector", "instrument"), storageClass="StampsBase", name="donutStamps", )
            Hide
            tjenness Tim Jenness added a comment -

            you need to add StampsBase to formatters.yaml in butler as well storageClasses.yaml.

            Show
            tjenness Tim Jenness added a comment - you need to add StampsBase to formatters.yaml in butler as well storageClasses.yaml.
            Hide
            tjenness Tim Jenness added a comment -

            This change looks good to me. If you put the jira ticket in review I can formally review it. Please remember to run Jenkins before merging.

            Show
            tjenness Tim Jenness added a comment - This change looks good to me. If you put the jira ticket in review I can formally review it. Please remember to run Jenkins before merging.
            Hide
            jbkalmbach Bryce Kalmbach added a comment -

            Great. Thanks for the help! Running jenkins now.

            Show
            jbkalmbach Bryce Kalmbach added a comment - Great. Thanks for the help! Running jenkins now.
            Hide
            tjenness Tim Jenness added a comment -

            Looks good to me.

            Show
            tjenness Tim Jenness added a comment - Looks good to me.

              People

              Assignee:
              jbkalmbach Bryce Kalmbach
              Reporter:
              jbkalmbach Bryce Kalmbach
              Reviewers:
              Tim Jenness
              Watchers:
              Bryce Kalmbach, Morgan Schmitz, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.