Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Story Points:4
-
Epic Link:
-
Team:Data Release Production
Description
Move registry code involving datasets (and, as needed, runs and collections) into helper classes as the prototype. If possible, defer actually moving from monolithic to split dataset tables to DM-21766.
Attachments
Issue Links
- blocks
-
- Done
-
- Done
-
- Done
-
- Done
-
- Invalid
- is contained by
-
- Done
Activity
| Labels | gen3-middleware | gen2-deprecation-blocker gen3-middleware |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Description |
Following the pattern established in This should include different subclasses for: * the current monolothic (one dataset table) approach * a hybrid with both a thin table for all datasets and wider tables (with dimension links) for each dataset type * (at prototype level) a chain of nested instances, for use with multi-user registries An important question is whether the new classes should: # just use SQLAlchemy objects directly, given them a one-way composition relationship with Registry, but no way for Registry subclasses to specialize database operations; # have a back-pointer to their Registry, and delegate to it for all low-level database operations. {{DimensionRecordStorage}} currently does (1), but (2) would work better with the new insert-with-conflict-resolution method added on |
Move registry code involving datasets (and, as needed, runs and collections) into helper classes as [the prototype|per https://confluence.lsstcorp.org/display/DM/Architectural+Prototype+for+the+New+Gen3+Registry]. If possible, defer actually moving from monolithic to split dataset tables to |
| Story Points | 8 | 4 |
| Labels | gen2-deprecation-blocker gen3-middleware | gen2-deprecation-blocker gen3-middleware gen3-registry-incompatibility |
| Description |
Move registry code involving datasets (and, as needed, runs and collections) into helper classes as [the prototype|per https://confluence.lsstcorp.org/display/DM/Architectural+Prototype+for+the+New+Gen3+Registry]. If possible, defer actually moving from monolithic to split dataset tables to |
Move registry code involving datasets (and, as needed, runs and collections) into helper classes as [the prototype|https://confluence.lsstcorp.org/display/DM/Architectural+Prototype+for+the+New+Gen3+Registry]. If possible, defer actually moving from monolithic to split dataset tables to |
Andy Salnikov, here's another fairly large review for you, but I'm hoping it will at least be familiar given that it's a refactoring pattern I know I've asked you to review before. All changes are in daf_butler (so far; running Jenkins now but I don't expect other packages to break), and most commits are small cleanups or improvements that either set up or react to the three big ones, which:
- add ABCs for helper classes for datasets in Registry;
- provide default implementations for those;
- switch Registry over to using them.
Jira seems to picked up an irrelevant PR as well as the right one, which is https://github.com/lsst/daf_butler/pull/266.
| Reviewers | Andy Salnikov [ salnikov ] | |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
Looks OK, few comments on PR.
| Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
| Resolution | Done [ 10000 ] | |
| Status | Reviewed [ 10101 ] | Done [ 10002 ] |
Restarting work on this now. I currently plan to do
DM-21766on the same branch (except they'll actually be per-sets-of-dimensions tables, not per-dataset-type), but notDM-21794orDM-24432(which currently seem no harder to do later).