Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24660

Improve Gen3 schema backwards compatibility

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:

      • copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler; we've already moved the dimension config into the database, which is even better; and check that the manager classes are consistent.  Here, I'll just write the manager classes into the per-repo config to decouple them from the subject-to-change daf_butler defaults.
      • starting to increment the version number in the dimensions config;
      • recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;
      • recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;
      • adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph
      • recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format
      • making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers; rejected as a bad idea (see first comment).
      • start recording a version number in YAML export files.

      Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Remote Link This issue links to "Page (Confluence)" [ 24390 ]
            Hide
            jbosch Jim Bosch added a comment -

            I started working on this bullet:

            making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers.

            on DM-24613 (where I thought it might save work overall, with DM-21794 on the horizon).  After that attempt, I think it's not actually helpful for schema stability and is a distraction from more pressing work.  I've put what I've done so far on the tickets/DM-24660 branch, and I'll revisit what to do with it when I actually get to this ticket.

            Show
            jbosch Jim Bosch added a comment - I started working on this bullet: making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers. on DM-24613 (where I thought it might save work overall, with DM-21794 on the horizon).  After that attempt, I think it's not actually helpful for schema stability and is a distraction from more pressing work.  I've put what I've done so far on the tickets/ DM-24660 branch, and I'll revisit what to do with it when I actually get to this ticket.
            jbosch Jim Bosch made changes -
            Description The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;
             * starting to increment the version number in the dimensions config;
             * recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;
             * recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;
             * adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph
             * recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format
             * making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers.

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * -copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;- {color:#de350b}we've already moving the dimension config into the database, which is even better; on this ticket, I'll do the same for the manager classes{color}.
             * -starting to increment the version number in the dimensions config;-
             * -recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;-
             * -recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;-
             * -adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph-
             * -recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format-
             * -making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers;- {color:#de350b}rejected as a bad idea (see first comment).{color}
             * {color:#de350b}start recording a version number in YAML export files.{color}

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Story Points 4 1
            Hide
            jbosch Jim Bosch added a comment -

            Most of this is already done (see annotations and strikethroughs in the original ticket description), and what's left is pretty easy and still worth doing.  I'm moving the current tickets/DM-24660 code to u/jbosch/DM-24660/compound-keys to make room.

            Show
            jbosch Jim Bosch added a comment - Most of this is already done (see annotations and strikethroughs in the original ticket description), and what's left is pretty easy and still worth doing.  I'm moving the current tickets/ DM-24660 code to u/jbosch/ DM-24660 /compound-keys to make room.
            jbosch Jim Bosch made changes -
            Description The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * -copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;- {color:#de350b}we've already moving the dimension config into the database, which is even better; on this ticket, I'll do the same for the manager classes{color}.
             * -starting to increment the version number in the dimensions config;-
             * -recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;-
             * -recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;-
             * -adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph-
             * -recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format-
             * -making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers;- {color:#de350b}rejected as a bad idea (see first comment).{color}
             * {color:#de350b}start recording a version number in YAML export files.{color}

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * -copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;- {color:#de350b}we've already moved the dimension config into the database, which is even better; and check that the manager classes are consistent.  Here, I'll just write the manager classes into the per-repo config to decouple them from the subject-to-change daf_butler defaults{color}.
             * -starting to increment the version number in the dimensions config;-
             * -recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;-
             * -recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;-
             * -adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph-
             * -recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format-
             * -making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers;- {color:#de350b}rejected as a bad idea (see first comment).{color}
             * {color:#de350b}start recording a version number in YAML export files.{color}

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            Hide
            jbosch Jim Bosch added a comment -

            I realize this is two review requests at once, but this is a very small one, and (with DM-27397, In Review to Andy and Nate) the last schema-change one I've got on my milestone radar.

            PR is here.

             

            Show
            jbosch Jim Bosch added a comment - I realize this is two review requests at once, but this is a very small one, and (with DM-27397 , In Review to Andy and Nate) the last schema-change one I've got on my milestone radar. PR is here .  
            jbosch Jim Bosch made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            tjenness Tim Jenness added a comment -

            Looks ok.

            Show
            tjenness Tim Jenness added a comment - Looks ok.
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-27033 [ DM-27033 ]
            jbosch Jim Bosch made changes -
            Epic Link DM-26786 [ 439736 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Tim Jenness
              Watchers:
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.