Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler, sphgeom
-
Story Points:2
-
Team:Architecture
-
Urgent?:No
Description
I've run into an issue when trying to import the Gen 3 portions of ap_verify_ci_hits2015 and ap_verify_ci_cosmos_pdr2. Both datasets' export files were created using the call
with butler.export(directory=<dir>, format="yaml") as contents:
|
contents.saveDatasets(butler.registry.queryDatasets(datasetType=..., collections=..., expand=True))
|
This creates patch/tract nodes in the export.yaml that look like:
- type: dimension
|
element: tract
|
records:
|
- skymap: deepCoadd_skyMap
|
id: 0
|
region: !!python/object/new:lsst.sphgeom.convexPolygon.ConvexPolygon
|
state: !!binary |
|
cFXvAKxKVuq/RvRf2oeC4T8FPPNfpn/DvyDNA7Oynu2/OLzXNgIs1j9xm1N9pn/Dv8YNITrW4O2/
|
YJqZWJtv1j8ULM5xUauyPz0LuTZumOq/F3pwY1Sk4T/+N9NnUauyPw==
|
When I try to import the export.yaml, I get the following stack trace:
File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/_butler.py", line 1339, in import_
|
doImport(stream)
|
File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/_butler.py", line 1332, in doImport
|
backend = BackendClass(importStream, self.registry)
|
File "/scratch/krzys001/daf_butler/python/lsst/daf/butler/core/repoTransfers.py", line 396, in __init__
|
wrapper = yaml.safe_load(stream)
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/__init__.py", line 162, in safe_load
|
return load(stream, SafeLoader)
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/__init__.py", line 114, in load
|
return loader.get_single_data()
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 51, in get_single_data
|
return self.construct_document(node)
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 60, in construct_document
|
for dummy in generator:
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
|
value = self.construct_mapping(node)
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 218, in construct_mapping
|
return super().construct_mapping(node, deep=deep)
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 143, in construct_mapping
|
value = self.construct_object(value_node, deep=deep)
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 100, in construct_object
|
data = constructor(self, node)
|
File "/software/lsstsw/stack_20200515/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages/yaml/constructor.py", line 429, in construct_undefined
|
node.start_mark)
|
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:lsst.sphgeom.convexPolygon.ConvexPolygon'
|
in "/project/krzys001/ap_verify_ci_cosmos_pdr2/config/export.yaml", line 4163, column 13
|
Tim Jenness recommends adding serialization support to sphgeom that makes these types compatible with yaml.safe_load. This is related to DM-25448, but distinct in that there's no LookupError with the registry.
Attachments
Issue Links
Activity
Krzysztof Findeisen I added the regions and pixelizations that we use in daf_butler. I don't think I need to do anything in daf_butler itself for things to work. Of course a new export will be needed once we merge. The main question is how we actually represent these things in YAML.
For regions I use a single hex string of the encoded bytes.
For pixelization scheme I use a mapping with one key "level". I could also use a single integer (seemingly as a string) if that is preferred.
I haven't tried to add YAML for all the sphgeom classes.
In the end I used a "mapping" for both. I also realized that I can query the node for the class name so could simplify the pixelization implementation.
I'll take a quick look.