This is the second half of the big registry overhaul (the big part being
DM-17023). Plans have been described in detail at https://docs.google.com/presentation/d/1KxVmRN_8S4GskyGxEkeoX7tn5c8b9xViy73JU4dkXOA/edit?usp=sharing.
- Normalizing the dataset table into different tables for each dataset type. This should improve query performance and give us flexibility in how we store metadata associated with datasets (possibly including regions and timestamps that are currently restricted to dimensions).
- Restructuring the registry codebase towards supporting eventual chained-schema registries.
- Enabling bulk inserts of datasets during ingest. This will require changes to Datastore as well (particularly its relationship with Registry).
- Addressing performance and concurrency problems in our usage of transactions.
Whenever possible, I'll try to split this up into smaller tickets. The sheer size of
DM-17023 has become a problem of its own, though I'm not sure how much I could have split it up. Happily this ticket should have a much smaller effect on public interfaces, though it will still involve some broad breaking changes.