The need for a near-complete rewrite of the queries subpackage has been building for a long time; there are simply enough major changes (over)due that it makes more sense to start a new subpackage, copy/move what's worth keeping (the expression parsing and validation, at least) there, and work on the new system until it's ready to replace the old one.
- Making query result objects that are serializable (via pydantic) and that do not have SQLAlchemy objects embedded in them, so they can be used in RemoteRegistry (with transformation to SQLAlchemy done only just before execution in SqlRegistry).
- Making use of the not-really-new-anymore pairwise spatial-overlap tables, for performance, to address
DM-31583, and to remove the need for post-query spatial filtering.
- Adding support for user-uploaded data IDs (sets the stage for data ID files on the command-line and new QG-gen algorithms).
- Making spatial joins user-controllable, to finally really fix QG generation limitations in jointcal, FGCM, etc.
- Separating "dataset constraint subquery" logic from "dataset search query" logic, and ensuring the WHERE constraints appear in the optimal place in each (optimizes https://lsstc.slack.com/archives/C01FBUGM2CV/p1631220754411100?thread_ts=1631203913.392200&cid=C01FBUGM2CV)
- Finally making calibration lookups vectorizable.
- Making result objects compatible with the new container ABCs to be introduced on DM-30332.
- Adding frequently-requested support for "what dataset types are in these collections" queries.
This has been on my radar for a ~year, but it got preempted by the no-work-found problem and stalled by the introduction of RemoteRegistry, which torpedoed my original prototype. But I think I have a solid enough idea now to try again.