Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21451

Remove DatabaseDict and vectorize Datastore/Butler ingest APIs

    XMLWordPrintable

    Details

      Description

      From #dm-middleware on slack:

      I'd like to get Datastores using the new schema stuff I added recently (TableSpec/FieldSpec) for anything they want to keep in the registry database, and I want to stop using a dict interface, as it's hard to map _setitem_ to insert and update, especially for bulk operations.  I'm thinking of:

      • Removing DatabaseDict and DatabaseDictRecordBase entirely.
      • Add abstract methods to StoredDatastoreItemInfo to translate to/from dictionaries that correspond to records to be saved.
      • Moving the logic for inserting and retrieving StoredDatastoreItemInfo from the database directly into GenericDatastoreBase.  That would probably be _insert_info and _fetch_info methods that replace _info_to_record and _record_to_info.
      • Overriding _insert_info and _fetch_info in InMemoryDatastore to save into a dict instead of letting GenericDatastoreBase's implementations write to the database.

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Epic Link DM-21254 [ 414685 ]
            jbosch Jim Bosch made changes -
            Link This issue is child task of DM-21231 [ DM-21231 ]
            jbosch Jim Bosch made changes -
            Description From #dm-middleware on slack:
            {quote}I'd like to get Datastores using the new schema stuff I added recently (TableSpec/FieldSpec) for anything they want to keep in the registry database, and I want to stop using a dict interface, as it's hard to map __setitem__ to insert and update, especially for bulk operations.  I'm thinking of:
             * Removing DatabaseDict and DatabaseDictRecordBase entirely.
             * Add abstract methods to StoredDatastoreItemInfo to translate to/from dictionaries that correspond to records to be saved.
             * Moving the logic for inserting and retrieving StoredDatastoreItemInfo from the database directly into GenericDatastoreBase.  That would probably be _insert_info and _fetch_info methods that replace _info_to_record and _record_to_info.
             * Overriding _insert_info and _fetch_info in InMemoryDatastore to save into a dict instead of letting GenericDatastoreBase's implementations write to the database.{quote}
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 22046 ]
            jbosch Jim Bosch made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jbosch Jim Bosch made changes -
            Link This issue is child task of DM-21231 [ DM-21231 ]
            jbosch Jim Bosch made changes -
            Link This issue is contained by DM-21231 [ DM-21231 ]
            jbosch Jim Bosch made changes -
            Link This issue contains DM-21794 [ DM-21794 ]
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Tim Jenness
              Watchers:
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.