Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24660

Improve Gen3 schema backwards compatibility

    XMLWordPrintable

Details

    • 1
    • Data Release Production
    • No

    Description

      The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:

      • copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler; we've already moved the dimension config into the database, which is even better; and check that the manager classes are consistent.  Here, I'll just write the manager classes into the per-repo config to decouple them from the subject-to-change daf_butler defaults.
      • starting to increment the version number in the dimensions config;
      • recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;
      • recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;
      • adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph
      • recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format
      • making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers; rejected as a bad idea (see first comment).
      • start recording a version number in YAML export files.

      Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            I started working on this bullet:

            making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers.

            on DM-24613 (where I thought it might save work overall, with DM-21794 on the horizon).  After that attempt, I think it's not actually helpful for schema stability and is a distraction from more pressing work.  I've put what I've done so far on the tickets/DM-24660 branch, and I'll revisit what to do with it when I actually get to this ticket.

            jbosch Jim Bosch added a comment - I started working on this bullet: making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers. on DM-24613 (where I thought it might save work overall, with DM-21794 on the horizon).  After that attempt, I think it's not actually helpful for schema stability and is a distraction from more pressing work.  I've put what I've done so far on the tickets/ DM-24660 branch, and I'll revisit what to do with it when I actually get to this ticket.
            jbosch Jim Bosch added a comment -

            Most of this is already done (see annotations and strikethroughs in the original ticket description), and what's left is pretty easy and still worth doing.  I'm moving the current tickets/DM-24660 code to u/jbosch/DM-24660/compound-keys to make room.

            jbosch Jim Bosch added a comment - Most of this is already done (see annotations and strikethroughs in the original ticket description), and what's left is pretty easy and still worth doing.  I'm moving the current tickets/ DM-24660 code to u/jbosch/ DM-24660 /compound-keys to make room.
            jbosch Jim Bosch added a comment -

            I realize this is two review requests at once, but this is a very small one, and (with DM-27397, In Review to Andy and Nate) the last schema-change one I've got on my milestone radar.

            PR is here.

             

            jbosch Jim Bosch added a comment - I realize this is two review requests at once, but this is a very small one, and (with DM-27397 , In Review to Andy and Nate) the last schema-change one I've got on my milestone radar. PR is here .  
            tjenness Tim Jenness added a comment -

            Looks ok.

            tjenness Tim Jenness added a comment - Looks ok.

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Tim Jenness
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.