Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31725

Rewrite butler registry.queries subpackage

    XMLWordPrintable

    Details

      Description

      The need for a near-complete rewrite of the queries subpackage has been building for a long time; there are simply enough major changes (over)due that it makes more sense to start a new subpackage, copy/move what's worth keeping (the expression parsing and validation, at least) there, and work on the new system until it's ready to replace the old one.

      Motivations include:

      • Making query result objects that are serializable (via pydantic) and that do not have SQLAlchemy objects embedded in them, so they can be used in RemoteRegistry (with transformation to SQLAlchemy done only just before execution in SqlRegistry).
      • Making use of the not-really-new-anymore pairwise spatial-overlap tables, for performance, to address DM-31583, and to remove the need for post-query spatial filtering.
      • Adding support for user-uploaded data IDs (sets the stage for data ID files on the command-line and new QG-gen algorithms).
      • Making spatial joins user-controllable, to finally really fix QG generation limitations in jointcal, FGCM, etc.
      • Separating "dataset constraint subquery" logic from "dataset search query" logic, and ensuring the WHERE constraints appear in the optimal place in each (optimizes https://lsstc.slack.com/archives/C01FBUGM2CV/p1631220754411100?thread_ts=1631203913.392200&cid=C01FBUGM2CV)
      • Finally making calibration lookups vectorizable.
      • Making result objects compatible with the new container ABCs to be introduced on DM-30332.
      • Adding frequently-requested support for "what dataset types are in these collections" queries.

      This has been on my radar for a ~year, but it got preempted by the no-work-found problem and stalled by the introduction of RemoteRegistry, which torpedoed my original prototype.  But I think I have a solid enough idea now to try again.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30438 Add support for uploading data IDs to temporary tables and vectorize data ID expansion

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31705 Make skypix dimensions usable in regular input, output, and quantum data IDs

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34838 Finish implementing dimension "populated_by" hooks and use them in default config

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34888 add a butler command to list a calibration collection

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-37409 Add calibration-collection temporal joins to QueryBuilder

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27660 Materialize dimension spatial relationships and overhaul query system, step 2

          • In Progress
          is parent task of

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36108 Move daf_butler's Ellipsis typing workaround to utils

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36111 Miscellaneous fixes and minor improvements to registry support classes

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36174 Pre-daf_relation query system refactoring

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36313 Overhaul registry dataset type and collection wildcards

          • Done
          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-878 Minor butler query API changes

          • Adopted

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-890 Add daf_relation as a daf_butler dependency

          • Implemented
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-37855 Sorting of dimension records no longer allows order by ID

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-37868 Remove undesirable defensiveness in Registry.findDatasets and fix query truncation bug

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-37938 Additional fixes for query spatial contraints

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-38943 Guard against invalid calls to count() in butler query CLI

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-32403 Support ORDER BY and LIMIT in registry query methods

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-37450 Respect dataset type storage class in registry query methods

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33621 Support reading dataIds from external file

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34263 Discuss Prompt Processing needs from middleware

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            8 older comments
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            I'd be super-amazed myself I was planning to work on schema migrations today, so there is little chance I can finish big review by tomorrow.

            Show
            salnikov Andy Salnikov added a comment - I'd be super-amazed myself I was planning to work on schema migrations today, so there is little chance I can finish big review by tomorrow.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Thanks for the hand-holding: I've merged daf_relation, tagged it w.2022.50, added the dependency to daf_butler's pyproject.toml, and the ctrl_mpexec PR build now succeeds. I think that means the PyPI stuff at least mostly worked.

            Show
            jbosch Jim Bosch added a comment - Thanks for the hand-holding: I've merged daf_relation, tagged it w.2022.50, added the dependency to daf_butler's pyproject.toml, and the ctrl_mpexec PR build now succeeds. I think that means the PyPI stuff at least mostly worked.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            PyPI: https://pypi.org/project/lsst-daf-relation/

            Next time you are fiddling with the package you need to fix the lack of readme content.

            Show
            tjenness Tim Jenness added a comment - PyPI: https://pypi.org/project/lsst-daf-relation/ Next time you are fiddling with the package you need to fix the lack of readme content.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Between Tim and me all packages have been reviewed now, marking the ticket as reviewed.

            Show
            salnikov Andy Salnikov added a comment - Between Tim and me all packages have been reviewed now, marking the ticket as reviewed.
            Hide
            Permalink
            jbosch Jim Bosch added a comment - - edited

            I believe I've now addressed all review comments and the GitHub Actions checks are green (including codecov, thanks to review-inspired new unit test nevermind, it's still ever so slightly red; just took a while to come in, I guess). I've kicked off a final Jenkins run to check that my rebasing didn't break anything, and will merge tomorrow morning if that succeeds.

            Show
            jbosch Jim Bosch added a comment - - edited I believe I've now addressed all review comments and the GitHub Actions checks are green ( including codecov, thanks to review-inspired new unit test nevermind, it's still ever so slightly red; just took a while to come in, I guess). I've kicked off a final Jenkins run to check that my rebasing didn't break anything, and will merge tomorrow morning if that succeeds.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Andy Salnikov, Tim Jenness
              Watchers:
              Andy Salnikov, Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.