Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24660

Improve Gen3 schema backwards compatibility

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:

      • copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler; we've already moved the dimension config into the database, which is even better; and check that the manager classes are consistent.  Here, I'll just write the manager classes into the per-repo config to decouple them from the subject-to-change daf_butler defaults.
      • starting to increment the version number in the dimensions config;
      • recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;
      • recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;
      • adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph
      • recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format
      • making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers; rejected as a bad idea (see first comment).
      • start recording a version number in YAML export files.

      Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Tim Jenness
              Watchers:
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.