Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21231

Refactor Registry handling of dataset and associated tables

    XMLWordPrintable

    Details

    • Story Points:
      0
    • Team:
      Data Release Production

      Description

      This is the second half of the big registry overhaul (the big part being DM-17023).  Plans have been described in detail at https://docs.google.com/presentation/d/1KxVmRN_8S4GskyGxEkeoX7tn5c8b9xViy73JU4dkXOA/edit?usp=sharing.

       

      Goals include:

      • Normalizing the dataset table into different tables for each dataset type.  This should improve query performance and give us flexibility in how we store metadata associated with datasets (possibly including regions and timestamps that are currently restricted to dimensions).
      • Restructuring the registry codebase towards supporting eventual chained-schema registries.
      • Enabling bulk inserts of datasets during ingest.  This will require changes to Datastore as well (particularly its relationship with Registry).
      • Addressing performance and concurrency problems in our usage of transactions.

      Whenever possible, I'll try to split this up into smaller tickets.  The sheer size of DM-17023 has become a problem of its own, though I'm not sure how much I could have split it up.  Happily this ticket should have a much smaller effect on public interfaces, though it will still involve some broad breaking changes.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21907 Implement multi-user Registries

          • To Do
          contains

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21448 Clean up DatasetRef comparisons and immutability

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21451 Remove DatabaseDict and vectorize Datastore/Butler ingest APIs

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21764 Better encapsulate dataset storage in Registry

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21766 Add per-dataset-type tables to Registry

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21768 Vectorize dataset insert API

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21795 Rework Registry provenance objects to match prototype

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21849 Make runs a type of collection

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24432 Add CALIBRATION collections and remove the calibration_label dimension

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24612 Add indexes to dataset_collection tables

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24614 Move dataset_location tables into manager/storage hierarchy

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21770 Vectorize dataset insert implementations

          • Invalid
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21201 Research cross-database approach to inserts with custom conflict resolution

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21203 Research cross-database approach to bulk inserts returning autoincrement values

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17023 Refactor the Dimensions and query system

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22487 Prototype Registry architecture for schema changes and (eventually) multiple layers

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17154 Move Registry schema definition from YAML to Python

          • Invalid
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue is blocked by DM-21021 [ DM-21021 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-21023 [ DM-21023 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-5309 [ DM-5309 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-21021 [ DM-21021 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-21023 [ DM-21023 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-5309 [ DM-5309 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-21201 [ DM-21201 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-17023 [ DM-17023 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-21203 [ DM-21203 ]
            jbosch Jim Bosch made changes -
            Link This issue is parent task of DM-21448 [ DM-21448 ]
            jbosch Jim Bosch made changes -
            Link This issue is parent task of DM-21451 [ DM-21451 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-17154 [ DM-17154 ]
            jbosch Jim Bosch made changes -
            Story Points 15 0
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21764 [ DM-21764 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21766 [ DM-21766 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21770 [ DM-21770 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21768 [ DM-21768 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21773 [ DM-21773 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21774 [ DM-21774 ]
            jbosch Jim Bosch made changes -
            Link This issue is parent task of DM-21448 [ DM-21448 ]
            jbosch Jim Bosch made changes -
            Link This issue is parent task of DM-21451 [ DM-21451 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21448 [ DM-21448 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21451 [ DM-21451 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21795 [ DM-21795 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21849 [ DM-21849 ]
            jbosch Jim Bosch made changes -
            Link This issue blocks DM-21907 [ DM-21907 ]
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-22487 [ DM-22487 ]
            tjenness Tim Jenness made changes -
            Labels gen2-deprecation-blocker gen3-middleware gen2-deprecation-blocker gen3-middleware gen3-registry-incompatibility
            jbosch Jim Bosch made changes -
            Link This issue contains DM-24612 [ DM-24612 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-24613 [ DM-24613 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-24614 [ DM-24614 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21773 [ DM-21773 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21774 [ DM-21774 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-24432 [ DM-24432 ]
            gruendl Robert Gruendl [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25770 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-24613 [ DM-24613 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.