Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29412

Update StampsBase so that it can serve as StorageClass for subclasses

    XMLWordPrintable

    Details

      Description

      For ts_wep we want to create a new subclass based upon the StampsBase class in meas_algorithms. To do this we are going to change StampsBase `readFits` and `writeFits` methods to allow this according to the prescription Tim Jenness described on slack in [this thread|https://lsstc.slack.com/archives/C01FBUGM2CV/p1616625257255300.] We will then have to add StampsBase to the StorageClass.yaml file in daf_butler.

        Attachments

          Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. DM-29531 Update IsrCalib so it can serve as StorageClass for subclasses

          • Done

            Activity

            Ascending order - Click to sort in descending order
            5 older comments
            Hide
            Permalink
            jbkalmbach Bryce Kalmbach added a comment -

            Yeah, I have a full traceback here:

             

            Process task-5:
            Traceback (most recent call last):
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 659, in _determine_put_formatter_location
            formatter = self.formatterFactory.getFormatter(ref,
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 751, in getFormatter
            _, formatter = self.getFormatterWithMatch(entity, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 724, in getFormatterWithMatch
            matchKey, formatter = self._mappingFactory.getFromRegistryWithMatch(names, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 188, in getFromRegistryWithMatch
            key, cls, registry_kwargs = self.getClassFromRegistryWithMatch(targetClasses)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 134, in getClassFromRegistryWithMatch
            raise KeyError(f"Unable to find item in registry with key{plural}: {msg}")
            KeyError: "Unable to find item in registry with keys: donutStamps ({'instrument': 'LSSTCam-PhoSim'}), band+instrument+detector+physical_filter+exposure ({'instrument': 'LSSTCam-PhoSim'}), StampsBase ({'instrument': 'LSSTCam-PhoSim'}), donutStamps, band+instrument+detector+physical_filter+exposure, StampsBase"

            The above exception was the direct cause of the following exception:

            Traceback (most recent call last):
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
            self.run()
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 108, in run
            self._target(*self._args, **self._kwargs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 127, in _executeJob
            quantumExecutor.execute(taskDef, quantum, butler)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute
            self.runQuantum(task, quantum, taskDef, butler)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 274, in runQuantum
            task.runQuantum(butlerQC, inputRefs, outputRefs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/pipelineTask.py", line 156, in runQuantum
            butlerQC.put(outputs, outputRefs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 198, in put
            self._put(valuesAttribute, refs)
            File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 98, in _put
            butler.put(value, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner
            return func(self, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/_butler.py", line 851, in put
            self.datastore.put(obj, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner
            return func(self, *args, **kwargs)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 1461, in put
            storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 952, in _write_in_memory_to_artifact
            location, formatter = self._prepare_for_put(inMemoryDataset, ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 627, in _prepare_for_put
            return self._determine_put_formatter_location(ref)
            File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 664, in _determine_put_formatter_location
            raise DatasetTypeNotSupportedError(f"Unable to find formatter for {ref} in datastore "
            lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError: Unable to find formatter for donutStamps@{instrument: 'LSSTCam-PhoSim', detector: 90, exposure: 9006001, ...}, sc=StampsBase] (id=21) in datastore FileDatastore@<butlerRoot>

             

            In the task I wrote the connections class has this to setup the donut stamps:

             

            donutStamps = connectionTypes.Output(
            doc="Donut Postage Stamp Images",
            dimensions=("exposure", "detector", "instrument"),
            storageClass="StampsBase",
            name="donutStamps",
            )

            Show
            jbkalmbach Bryce Kalmbach added a comment - Yeah, I have a full traceback here:   Process task-5: Traceback (most recent call last): File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 659, in _determine_put_formatter_location formatter = self.formatterFactory.getFormatter(ref, File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 751, in getFormatter _, formatter = self.getFormatterWithMatch(entity, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/formatter.py", line 724, in getFormatterWithMatch matchKey, formatter = self._mappingFactory.getFromRegistryWithMatch(names, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 188, in getFromRegistryWithMatch key, cls, registry_kwargs = self.getClassFromRegistryWithMatch(targetClasses) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/mappingFactory.py", line 134, in getClassFromRegistryWithMatch raise KeyError(f"Unable to find item in registry with key{plural}: {msg}") KeyError: "Unable to find item in registry with keys: donutStamps ({'instrument': 'LSSTCam-PhoSim'}), band+instrument+detector+physical_filter+exposure ({'instrument': 'LSSTCam-PhoSim'}), StampsBase ({'instrument': 'LSSTCam-PhoSim'}), donutStamps, band+instrument+detector+physical_filter+exposure, StampsBase" The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.4.1/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 127, in _executeJob quantumExecutor.execute(taskDef, quantum, butler) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute self.runQuantum(task, quantum, taskDef, butler) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-18-gaa77eba+59b91001fb/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 274, in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/pipelineTask.py", line 156, in runQuantum butlerQC.put(outputs, outputRefs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 198, in put self._put(valuesAttribute, refs) File "/astro/store/epyc/projects/lsst_comm/stack_w_2021_07/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-8-gfbe0b4b+411cd868f8/python/lsst/pipe/base/butlerQuantumContext.py", line 98, in _put butler.put(value, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner return func(self, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/_butler.py", line 851, in put self.datastore.put(obj, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/core/utils.py", line 265, in inner return func(self, *args, **kwargs) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 1461, in put storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 952, in _write_in_memory_to_artifact location, formatter = self._prepare_for_put(inMemoryDataset, ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 627, in _prepare_for_put return self._determine_put_formatter_location(ref) File "/astro/store/epyc/users/brycek/Commissioning/aos/clean_setup/daf_butler/python/lsst/daf/butler/datastores/fileDatastore.py", line 664, in _determine_put_formatter_location raise DatasetTypeNotSupportedError(f"Unable to find formatter for {ref} in datastore " lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError: Unable to find formatter for donutStamps@{instrument: 'LSSTCam-PhoSim', detector: 90, exposure: 9006001, ...}, sc=StampsBase] (id=21) in datastore FileDatastore@<butlerRoot>   In the task I wrote the connections class has this to setup the donut stamps:   donutStamps = connectionTypes.Output( doc="Donut Postage Stamp Images", dimensions=("exposure", "detector", "instrument"), storageClass="StampsBase", name="donutStamps", )
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            you need to add StampsBase to formatters.yaml in butler as well storageClasses.yaml.

            Show
            tjenness Tim Jenness added a comment - you need to add StampsBase to formatters.yaml in butler as well storageClasses.yaml.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            This change looks good to me. If you put the jira ticket in review I can formally review it. Please remember to run Jenkins before merging.

            Show
            tjenness Tim Jenness added a comment - This change looks good to me. If you put the jira ticket in review I can formally review it. Please remember to run Jenkins before merging.
            Hide
            Permalink
            jbkalmbach Bryce Kalmbach added a comment -

            Great. Thanks for the help! Running jenkins now.

            Show
            jbkalmbach Bryce Kalmbach added a comment - Great. Thanks for the help! Running jenkins now.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Looks good to me.

            Show
            tjenness Tim Jenness added a comment - Looks good to me.

              People

              Assignee:
              jbkalmbach Bryce Kalmbach
              Reporter:
              jbkalmbach Bryce Kalmbach
              Reviewers:
              Tim Jenness
              Watchers:
              Bryce Kalmbach, Morgan Schmitz, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.