# Cannot import sphgeom objects in Gen 3

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
2
• Team:
Architecture
• Urgent?:
No

#### Description

I've run into an issue when trying to import the Gen 3 portions of ap_verify_ci_hits2015 and ap_verify_ci_cosmos_pdr2. Both datasets' export files were created using the call

 with butler.export(directory=, format="yaml") as contents:  contents.saveDatasets(butler.registry.queryDatasets(datasetType=..., collections=..., expand=True)) 

This creates patch/tract nodes in the export.yaml that look like:

 - type: dimension  element: tract  records:  - skymap: deepCoadd_skyMap  id: 0  region: !!python/object/new:lsst.sphgeom.convexPolygon.ConvexPolygon  state: !!binary |  cFXvAKxKVuq/RvRf2oeC4T8FPPNfpn/DvyDNA7Oynu2/OLzXNgIs1j9xm1N9pn/Dv8YNITrW4O2/  YJqZWJtv1j8ULM5xUauyPz0LuTZumOq/F3pwY1Sk4T/+N9NnUauyPw== 

When I try to import the export.yaml, I get the following stack trace:

  File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/_butler.py", line 1339, in import_  doImport(stream)  File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/_butler.py", line 1332, in doImport  backend = BackendClass(importStream, self.registry)  File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/core/repoTransfers.py", line 396, in __init__  wrapper = yaml.safe_load(stream)  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/__init__.py", line 162, in safe_load  return load(stream, SafeLoader)  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/__init__.py", line 114, in load  return loader.get_single_data()  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 51, in get_single_data  return self.construct_document(node)  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 60, in construct_document  for dummy in generator:  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 413, in construct_yaml_map  value = self.construct_mapping(node)  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 218, in construct_mapping  return super().construct_mapping(node, deep=deep)  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 143, in construct_mapping  value = self.construct_object(value_node, deep=deep)  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 100, in construct_object  data = constructor(self, node)  File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 429, in construct_undefined  node.start_mark) yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:lsst.sphgeom.convexPolygon.ConvexPolygon'  in "/project/krzys001/ap_verify_ci_cosmos_pdr2/config/export.yaml", line 4163, column 13 

Tim Jenness recommends adding serialization support to sphgeom that makes these types compatible with yaml.safe_load. This is related to DM-25448, but distinct in that there's no LookupError with the registry.

#### Activity

Hide
Tim Jenness added a comment -

I'll take a quick look.

Show
Tim Jenness added a comment - I'll take a quick look.
Hide
Tim Jenness added a comment -

Krzysztof Findeisen I added the regions and pixelizations that we use in daf_butler. I don't think I need to do anything in daf_butler itself for things to work. Of course a new export will be needed once we merge. The main question is how we actually represent these things in YAML.

For regions I use a single hex string of the encoded bytes.
For pixelization scheme I use a mapping with one key "level". I could also use a single integer (seemingly as a string) if that is preferred.

I haven't tried to add YAML for all the sphgeom classes.

Show
Tim Jenness added a comment - Krzysztof Findeisen I added the regions and pixelizations that we use in daf_butler. I don't think I need to do anything in daf_butler itself for things to work. Of course a new export will be needed once we merge. The main question is how we actually represent these things in YAML. For regions I use a single hex string of the encoded bytes. For pixelization scheme I use a mapping with one key "level". I could also use a single integer (seemingly as a string) if that is preferred. I haven't tried to add YAML for all the sphgeom classes.
Hide
Tim Jenness added a comment -

In the end I used a "mapping" for both. I also realized that I can query the node for the class name so could simplify the pixelization implementation.

Show
Tim Jenness added a comment - In the end I used a "mapping" for both. I also realized that I can query the node for the class name so could simplify the pixelization implementation.

#### People

Assignee:
Tim Jenness
Reporter:
Krzysztof Findeisen
Reviewers:
Krzysztof Findeisen
Watchers:
Jim Bosch, Krzysztof Findeisen, Tim Jenness