Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15336

Decouple DataUnitRegistry from Schema

    Details

    • Story Points:
      10
    • Sprint:
      DRP F18-4
    • Team:
      Data Release Production

      Description

      In order to use DataUnit objects inside DatasetType and DataId objects on DM-15034, we need to make it possible to construct a DataUnitRegistry without having a Registry instance.  In many ways that makes DataUnitRegistry analogous to StorageClassFactory, and hence we should follow the same approach: make it an append-only singleton.

      This is complicated by the fact that Schema and DataUnitRegistry are closely intertwined; DataUnit objects actually currently have some SQLAlchemy objects attached to them.  We'll have to remove those as part of this ticket, and instead attach them to Schema in a way that allows it to retrieve them with DataUnit objects or names as keys.

      The recommended implementation plan is roughly:

      • Move the "tables" sections out of the individual DataUnit sections in schema.yaml into the main "tables" section at the end; this should break at most a tiny bit of the parsing logic for that configuration (and possibly nothing).
      • Remove the "foreignKey" sections nested under "link" sections in the yaml; instead generate these programmatically.
      • Split the DataUnit content (now free of any SQL-specific stuff) into its own yaml file; adjust the levels of both dataUnits.yaml and schema.yaml appropriately (e.g. remove the now-superfluous "tables" section heading?).
      • Change SchemaBuilder so SQLAlchemy objects are put somewhere in Schema where they can be looked up via DataUnit name or object, instead in the DataUnit objects themselves.  Fix whatever this breaks in SqlRegistry and SqlPreflight.

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment -

            merged to master

            Show
            nlust Nate Lust added a comment - merged to master
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            Pair-coded most of this yesterday and today. Only the split-up (of the dataUnits and dataUnitJoins sections) into a separate dataUnits.yaml file is still needed.

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - Pair-coded most of this yesterday and today. Only the split-up (of the dataUnits and dataUnitJoins sections) into a separate dataUnits.yaml file is still needed.

              People

              • Assignee:
                nlust Nate Lust
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, Nate Lust, Pim Schellart [X] (Inactive), Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel