Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21451

Remove DatabaseDict and vectorize Datastore/Butler ingest APIs

    XMLWordPrintable

    Details

      Description

      From #dm-middleware on slack:

      I'd like to get Datastores using the new schema stuff I added recently (TableSpec/FieldSpec) for anything they want to keep in the registry database, and I want to stop using a dict interface, as it's hard to map _setitem_ to insert and update, especially for bulk operations.  I'm thinking of:

      • Removing DatabaseDict and DatabaseDictRecordBase entirely.
      • Add abstract methods to StoredDatastoreItemInfo to translate to/from dictionaries that correspond to records to be saved.
      • Moving the logic for inserting and retrieving StoredDatastoreItemInfo from the database directly into GenericDatastoreBase.  That would probably be _insert_info and _fetch_info methods that replace _info_to_record and _record_to_info.
      • Overriding _insert_info and _fetch_info in InMemoryDatastore to save into a dict instead of letting GenericDatastoreBase's implementations write to the database.

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Epic Link DM-21254 [ 414685 ]
            jbosch Jim Bosch made changes -
            Link This issue is child task of DM-21231 [ DM-21231 ]
            jbosch Jim Bosch made changes -
            Description From #dm-middleware on slack:
            {quote}I'd like to get Datastores using the new schema stuff I added recently (TableSpec/FieldSpec) for anything they want to keep in the registry database, and I want to stop using a dict interface, as it's hard to map __setitem__ to insert and update, especially for bulk operations.  I'm thinking of:
             * Removing DatabaseDict and DatabaseDictRecordBase entirely.
             * Add abstract methods to StoredDatastoreItemInfo to translate to/from dictionaries that correspond to records to be saved.
             * Moving the logic for inserting and retrieving StoredDatastoreItemInfo from the database directly into GenericDatastoreBase.  That would probably be _insert_info and _fetch_info methods that replace _info_to_record and _record_to_info.
             * Overriding _insert_info and _fetch_info in InMemoryDatastore to save into a dict instead of letting GenericDatastoreBase's implementations write to the database.{quote}
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 22046 ]
            Hide
            jbosch Jim Bosch added a comment -

            Tim Jenness, I know you're traveling, but you're still pretty clearly the best candidate for this review.  No rush - I'll need to land some other changes before we can gain much advantage from these, and at least the first few of those will be quite orthogonal.

            Show
            jbosch Jim Bosch added a comment - Tim Jenness , I know you're traveling, but you're still pretty clearly the best candidate for this review.  No rush - I'll need to land some other changes before we can gain much advantage from these, and at least the first few of those will be quite orthogonal.
            jbosch Jim Bosch made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jbosch Jim Bosch made changes -
            Link This issue is child task of DM-21231 [ DM-21231 ]
            jbosch Jim Bosch made changes -
            Link This issue is contained by DM-21231 [ DM-21231 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21794 [ DM-21794 ]
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            Hide
            jbosch Jim Bosch added a comment -

            Tim Jenness, ping: review request may have gone out while Jira wasn't sending them.

            Show
            jbosch Jim Bosch added a comment - Tim Jenness , ping: review request may have gone out while Jira wasn't sending them.
            Hide
            tjenness Tim Jenness added a comment -

            In my defense I started doing the review five minutes before your comment turned up

            Show
            tjenness Tim Jenness added a comment - In my defense I started doing the review five minutes before your comment turned up
            Hide
            tjenness Tim Jenness added a comment -

            Looks good. I have some questions on the PR.

            Show
            tjenness Tim Jenness added a comment - Looks good. I have some questions on the PR.
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Tim Jenness
              Watchers:
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.