Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler, pipe_base
-
Story Points:6
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
I am getting this failure with w_2022_23. The same YAML worked on _22.
lsst.ctrl.bps.pre_transform ERROR: Completed reading quantum graph: Took 1.5278 seconds
|
lsst.ctrl.bps.drivers ERROR: Acquire stage completed: Took 23.8714 seconds; current memory usage: 0.329 Gibyte, delta: 0.116 Gibyte, peak delta: 0.116 Gibyte
|
lsst.ctrl.bps.drivers ERROR: Completed entire submission process: Took 28.7855 seconds; current memory usage: 0.329 Gibyte, delta: 0.210 Gibyte, peak delta: 0.210 Gibyte
|
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/cli/cmd/commands.py", line 84, in submit
|
submit_driver(*args, **kwargs)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/drivers.py", line 387, in submit_driver
|
wms_workflow_config, wms_workflow = prepare_driver(config_file, **kwargs)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/drivers.py", line 338, in prepare_driver
|
generic_workflow_config, generic_workflow = transform_driver(config_file, **kwargs)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/drivers.py", line 289, in transform_driver
|
config, clustered_qgraph = cluster_qgraph_driver(config_file, **kwargs)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/drivers.py", line 244, in cluster_qgraph_driver
|
config, qgraph = acquire_qgraph_driver(config_file, **kwargs)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/drivers.py", line 217, in acquire_qgraph_driver
|
qgraph_file, qgraph, execution_butler_dir = acquire_quantum_graph(config, out_prefix=submit_path)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/utils/g894b03d5c6+82f6db4df0/python/lsst/utils/timer.py", line 345, in timeMethod_wrapper
|
res = func(self, *args, **keyArgs)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/pre_transform.py", line 109, in acquire_quantum_graph
|
qgraph = read_quantum_graph(qgraph_filename)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/ctrl_bps/g18e6bc4727+f60974356c/python/lsst/ctrl/bps/pre_transform.py", line 213, in read_quantum_graph
|
qgraph = QuantumGraph.loadUri(qgraph_filename, DimensionUniverse())
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/pipe_base/g0ca2d2c848+97368ef238/python/lsst/pipe/base/graph/graph.py", line 808, in loadUri
|
qgraph = loader.load(universe, nodes, graphID)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/pipe_base/g0ca2d2c848+97368ef238/python/lsst/pipe/base/graph/_loadHelpers.py", line 264, in load
|
return self.deserializer.constructGraph(nodeSet, _readBytes, universe)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/pipe_base/g0ca2d2c848+97368ef238/python/lsst/pipe/base/graph/_versionDeserializers.py", line 546, in constructGraph
|
qnode = QuantumNode.from_simple(nodeDeserialized, loadedTaskDef, universe, recontitutedDimensions)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/pipe_base/g0ca2d2c848+97368ef238/python/lsst/pipe/base/graph/quantumNode.py", line 141, in from_simple
|
quantum=Quantum.from_simple(
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/daf_butler/g31b4056c07+59a6ed3a8e/python/lsst/daf/butler/core/quantum.py", line 449, in from_simple
|
rebuiltDatasetRef = _reconstructDatasetRef(
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/daf_butler/g31b4056c07+59a6ed3a8e/python/lsst/daf/butler/core/quantum.py", line 64, in _reconstructDatasetRef
|
reconstructedDim = DimensionRecord.from_simple(tmpSerialized, universe=universe)
|
File "/software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/daf_butler/g31b4056c07+59a6ed3a8e/python/lsst/daf/butler/core/dimensions/_records.py", line 379, in from_simple
|
record_model = record_model_cls(**simple.record)
|
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
|
pydantic.error_wrappers.ValidationError: 1 validation error for SpecificSerializedDimensionRecordInstrument
|
visit_system
|
This problem was fixed temporarily in DM-35090. The fundamental problem is that quantum graphs do not know the universe they are associated with. This ticket is now to store the universe in the graph.
Attachments
Issue Links
Activity
Once the graph contains the universe:
- The dotTools package can get the universe directly.
- pipetask run can validate that the universe in the graph matches the universe in the butler it is using.
- ctrl_bps can also do an early validation against the butler.
I see that the rc2_subset package uses a naked DimensionUniverse in a helper script even though it has a Butler.
The Skymap package uses DimensionUniverse in baseSkyMap.pack_data_id – this is a transitional API to support gen2 to gen3 migration. It is only used in coadd_utils getGen3CoaddExposureId. That is used in 6 files across meas_base, pipe_drivers, and pipe_tasks. There is one usage in makeIdFactory but mostly runDataRef. It's all gen2 compatibility code that is no longer called.
Looks good. I think we did not manage to avoid using global DimensionUniverse in dotTools.pipeline2dot() method as it does not have a graph, only Pipeline. I think for that method it should be OK to pass Butler's universe, but probably in all cases when that method is used Butler's universe should be the same as global one?
DM-35494 is waiting on this to merge.
Paul Price
added a comment - DM-35494 is waiting on this to merge. Paul Price / Matthias Wittgen I think you should be unblocked for pipe_base merges now.
Tests are usually fine using the default universe. We've only spotted this problem because the default universe and the most commonly used universe are now incompatible. We clearly need to solve the general problem of graphs not knowing their universe.