Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24660

Improve Gen3 schema backwards compatibility

    XMLWordPrintable

Details

    • 1
    • Data Release Production
    • No

    Description

      The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:

      • copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler; we've already moved the dimension config into the database, which is even better; and check that the manager classes are consistent.  Here, I'll just write the manager classes into the per-repo config to decouple them from the subject-to-change daf_butler defaults.
      • starting to increment the version number in the dimensions config;
      • recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;
      • recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;
      • adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph
      • recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format
      • making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers; rejected as a bad idea (see first comment).
      • start recording a version number in YAML export files.

      Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Remote Link This issue links to "Page (Confluence)" [ 24390 ]
            jbosch Jim Bosch made changes -
            Description The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;
             * starting to increment the version number in the dimensions config;
             * recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;
             * recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;
             * adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph
             * recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format
             * making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers.

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * -copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;- {color:#de350b}we've already moving the dimension config into the database, which is even better; on this ticket, I'll do the same for the manager classes{color}.
             * -starting to increment the version number in the dimensions config;-
             * -recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;-
             * -recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;-
             * -adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph-
             * -recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format-
             * -making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers;- {color:#de350b}rejected as a bad idea (see first comment).{color}
             * {color:#de350b}start recording a version number in YAML export files.{color}

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Story Points 4 1
            jbosch Jim Bosch made changes -
            Description The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * -copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;- {color:#de350b}we've already moving the dimension config into the database, which is even better; on this ticket, I'll do the same for the manager classes{color}.
             * -starting to increment the version number in the dimensions config;-
             * -recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;-
             * -recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;-
             * -adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph-
             * -recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format-
             * -making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers;- {color:#de350b}rejected as a bad idea (see first comment).{color}
             * {color:#de350b}start recording a version number in YAML export files.{color}

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            The registry refactor to delegate to polymorphic helper classes should allow us to make many different kinds of changes to the Registry implementation that affect the database schema without breaking our ability to read old repos with new code.  Actually making that happen requires a number of smaller changes, and there are many more that would further improve our backwards compatibility situation:
             * -copying (and effectively freezing) the Registry config subtrees for dimensions and manager classes into per-repo configs, instead of always pulling defaults from daf_butler;- {color:#de350b}we've already moved the dimension config into the database, which is even better; and check that the manager classes are consistent.  Here, I'll just write the manager classes into the per-repo config to decouple them from the subject-to-change daf_butler defaults{color}.
             * -starting to increment the version number in the dimensions config;-
             * -recording the dimensions version in the database somewhere, so we can easily check for consistency and raise more helpful exceptions when the config is inconsistent with the database;-
             * -recording the manager classes used in the database (for the same reason), and perhaps adding version numbers for these as well;-
             * -adding dimension version numbers or some other form of stability to the byte-encoded form of DimensionGraph-
             * -recording the names of dynamic tables in the static tables that manage them to reduce reliance on any particular name format-
             * -making it so Manager classes that add foreign key fields can use compound keys, further encapsulating DDL schemas inside the managers;- {color:#de350b}rejected as a bad idea (see first comment).{color}
             * {color:#de350b}start recording a version number in YAML export files.{color}

            Note that this ticket is about reducing the need for schema migration (or at least giving us more flexibility about when any particular repo is migrated).  We'll still need to have some way to do actual schema migrations to upgrade old repos when they can't just be thrown away.
            jbosch Jim Bosch made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-27033 [ DM-27033 ]
            jbosch Jim Bosch made changes -
            Epic Link DM-26786 [ 439736 ]

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Tim Jenness
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.