Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25786

Cannot import sphgeom objects in Gen 3

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      I've run into an issue when trying to import the Gen 3 portions of ap_verify_ci_hits2015 and ap_verify_ci_cosmos_pdr2. Both datasets' export files were created using the call

      with butler.export(directory=<dir>, format="yaml") as contents:
          contents.saveDatasets(butler.registry.queryDatasets(datasetType=..., collections=..., expand=True))
      

      This creates patch/tract nodes in the export.yaml that look like:

      - type: dimension
        element: tract
        records:
        - skymap: deepCoadd_skyMap
          id: 0
          region: !!python/object/new:lsst.sphgeom.convexPolygon.ConvexPolygon
            state: !!binary |
              cFXvAKxKVuq/RvRf2oeC4T8FPPNfpn/DvyDNA7Oynu2/OLzXNgIs1j9xm1N9pn/Dv8YNITrW4O2/
              YJqZWJtv1j8ULM5xUauyPz0LuTZumOq/F3pwY1Sk4T/+N9NnUauyPw==
      

      When I try to import the export.yaml, I get the following stack trace:

        File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/_butler.py", line 1339, in import_
          doImport(stream)
        File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/_butler.py", line 1332, in doImport
          backend = BackendClass(importStream, self.registry)
        File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/core/repoTransfers.py", line 396, in __init__
          wrapper = yaml.safe_load(stream)
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/__init__.py", line 162, in safe_load
          return load(stream, SafeLoader)
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/__init__.py", line 114, in load
          return loader.get_single_data()
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 51, in get_single_data
          return self.construct_document(node)
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 60, in construct_document
          for dummy in generator:
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
          value = self.construct_mapping(node)
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 218, in construct_mapping
          return super().construct_mapping(node, deep=deep)
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 143, in construct_mapping
          value = self.construct_object(value_node, deep=deep)
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 100, in construct_object
          data = constructor(self, node)
        File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 429, in construct_undefined
          node.start_mark)
      yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:lsst.sphgeom.convexPolygon.ConvexPolygon'
        in "/project/krzys001/ap_verify_ci_cosmos_pdr2/config/export.yaml", line 4163, column 13
      

      Tim Jenness recommends adding serialization support to sphgeom that makes these types compatible with yaml.safe_load. This is related to DM-25448, but distinct in that there's no LookupError with the registry.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Krzysztof Findeisen
              Watchers:
              Jim Bosch, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.