We have a lot of missing test coverage in the middleware packages that have let a lot of bugs slip through onto main recently, especially in execution butler, QuantumGraph generation, and corners of the butler query system exercised only by those. It's very hard to write complete unit tests for these components in the packages that implement, because the most complicated bits of logic involve interactions between our realistically-complicated data model and a realistically complicated pipeline. We've written many test utility suites with made-up data across several packages but none of them are really adequate (or easy to extend without breaking lots of existing tests).
But we also have a powerful tool we've barely used - the mock-execution system created on
DM-31253. We also have plenty of realistic test data in various repos that would collapse down to something pretty small if we exported just the database content and replaced files with tiny stubs.
My plan here is to create a new ci_middleware package with that kind of test data (probably in the form of an export YAML and library code to make a repo from it) and test scripts. It'd depend on on drp_pipe to get some real pipelines (AP pipelines are at present a subset of DRP from the middleware-complexity standpoint) and some real obs package (probably obs_subaru, since that's what will support the more complex pipelines and calibrations right now). It'll also have required dependencies on daf_butler, pipe_base, and obs_base, and an optional dependency on ctrl_bps and and particular BPS extension packages we want to test there, with those optional tests skipped if the appropriate packages are set up.
My first goal here is being able to test QG generation, but I definitely want to be able to test SingleQuantumExecutor against QBB and execution butler here as well, by running the real DRP pipelines in mock mode. Integrating testing involving different ResourcePath schemes was not something in mind, but I don't know much about setting up test environments for that, and I'm not opposed to the idea.
If the exported test data is on the large side, this might involve splitting of a testdata_middleware for that.