Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21451

Remove DatabaseDict and vectorize Datastore/Butler ingest APIs

    XMLWordPrintable

Details

    Description

      From #dm-middleware on slack:

      I'd like to get Datastores using the new schema stuff I added recently (TableSpec/FieldSpec) for anything they want to keep in the registry database, and I want to stop using a dict interface, as it's hard to map _setitem_ to insert and update, especially for bulk operations.  I'm thinking of:

      • Removing DatabaseDict and DatabaseDictRecordBase entirely.
      • Add abstract methods to StoredDatastoreItemInfo to translate to/from dictionaries that correspond to records to be saved.
      • Moving the logic for inserting and retrieving StoredDatastoreItemInfo from the database directly into GenericDatastoreBase.  That would probably be _insert_info and _fetch_info methods that replace _info_to_record and _record_to_info.
      • Overriding _insert_info and _fetch_info in InMemoryDatastore to save into a dict instead of letting GenericDatastoreBase's implementations write to the database.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            tjenness, I know you're traveling, but you're still pretty clearly the best candidate for this review.  No rush - I'll need to land some other changes before we can gain much advantage from these, and at least the first few of those will be quite orthogonal.

            jbosch Jim Bosch added a comment - tjenness , I know you're traveling, but you're still pretty clearly the best candidate for this review.  No rush - I'll need to land some other changes before we can gain much advantage from these, and at least the first few of those will be quite orthogonal.
            jbosch Jim Bosch added a comment -

            tjenness, ping: review request may have gone out while Jira wasn't sending them.

            jbosch Jim Bosch added a comment - tjenness , ping: review request may have gone out while Jira wasn't sending them.
            tjenness Tim Jenness added a comment -

            In my defense I started doing the review five minutes before your comment turned up

            tjenness Tim Jenness added a comment - In my defense I started doing the review five minutes before your comment turned up
            tjenness Tim Jenness added a comment -

            Looks good. I have some questions on the PR.

            tjenness Tim Jenness added a comment - Looks good. I have some questions on the PR.

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Tim Jenness
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.