added a comment - Andy Salnikov , I'm afraid I've got another big review for you, but Nate Lust got the last one and Tim Jenness has gotten a ton of little ones, so you're up.
I've updated the ticket description to reflect the scope that actually ended up here. I should also add a bit about what's not here. I haven't made downstream code in ctrl_mpexec (or anywhere else) use the new QuantumGraphBuilder directly yet, but since the old GraphBuilder interface delegates to the new code it is actually in use. Removing use of GraphBuilder will be DM-40441 .
Jenkins is still running and I want to do some new benchmarks now that this is fully done (preliminary ones are very promising), but I think it's not worth holding up the review for those. If Jenkins does spot something I'll make sure the changes appear as separate commits.
The big new additions are the QuantumGraphBuilder base class and its AllDimensionsQuantumGraphBuilder derived class, which provide the bulk of the new QG generation implementation. The quantum_graph_skeleton and prerequisite_helper modules provide helper classes for them. All of these are in pipe_base .
I've also added some small but critical new code in daf_butler to expand the query system a bit; this lets us do bulk queries for most prerequisite inputs, which yields a huge speedup for some pipelines.
The other two PRs are ctrl_mpexec and ci_middleware , and are just small changes to tests.
Test coverage here is hard to gauge, because most of the coverage is in ci_middleware , while most of the changes are in pipe_base . So I've manually combined coverage from all locations, and here's the result:
$ diff-cover coverage.xml
Diff: origin/main...HEAD, staged and unstaged changes
python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py (85.7%): Missing lines 152,192,230-231,263,266-268,303,309,312,318,320,326,468,477,499-502,511-518
python/lsst/pipe/base/prerequisite_helpers.py (79.3%): Missing lines 147,263,278,284,286,334,349,423-424,426,445-448,450,538-539,542,553,556,565,568,577,595,601,631,636-641,643,649-652,655
python/lsst/pipe/base/quantum_graph_builder.py (88.5%): Missing lines 178,180,183,185,194,364,499,503-508,514-515,517,519,524-525,528,532-533,540,549,556,624,687,692,850-851,895,933-934,956-957,959,1085-1090
python/lsst/pipe/base/quantum_graph_skeleton.py (94.9%): Missing lines 164,286-288,320,350,393
Total: 943 lines
Missing: 115 lines
The missing coverage falls into three categories:
exception-raises that are just short of assertions (e.g. checks for nonsensical pipelines) that I just don't think are worth working too hard to test;
the actual functionality that originally spurred the ticket (being able to raise NoWorkFound in adjustQuantum, and have that propagate to downstream nodes): I'm going to do a one-off test of this using the DM-37383 ticket branches tomorrow, and then after that merges I'll make sure it gets included in ci_middleware;
a lot of the code paths in PrerequisiteFinder are fallback code paths used by the QuantumGraphBuilder base class, for which its only concrete subclass ( AllDimensionsQuantumGraphBuilder ) provides optimized overrides. I'm not sure what to do about testing those.