Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30344

Add public registry property declaring the dataset_id type

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Currently we have two python types that can be used by a registry: UUID and int. Sometimes a caller needs to change what they do based on knowledge of this type. For example when ingesting raw files you want to use a UUID5 in UUID registry but unique int in integer registry. Being able to ask the registry what type it is using will simplify this logic.

      It needs to be able to do this without querying for datasets in order to get a DatasetRef to inspect the type in use there.

      A remote client/server registry will have to report the type in use by the registry the server is attached to.

      This ticket will clean up butler ingest and butler dataset transfer code to simplify some of the logic.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          I think I'm going to take care of this on a PREOPS-579 branch, because:

          • I want to add support for not trying to ingest already-ingested files to RawIngestTask in order to accomplish that ticket (I could query for and remove already-ingested files outside RawIngestTask instead, but I think it's useful functionality to have and probably easier to implement inside the task as well).
          • The try/except block currently used to test for UUID support will get in the way of the transaction handling I'll need to implement that new functionality.

          I am leaning towards

          Registry.supportsDatasetIdGenerationMode(mode: DatasetIdGenEnum) -> bool

          rather that something that is explicitly about integer vs UUID.

          Show
          jbosch Jim Bosch added a comment - I think I'm going to take care of this on a PREOPS-579 branch, because: I want to add support for not trying to ingest already-ingested files to RawIngestTask in order to accomplish that ticket (I could query for and remove already-ingested files outside RawIngestTask instead, but I think it's useful functionality to have and probably easier to implement inside the task as well). The try/except block currently used to test for UUID support will get in the way of the transaction handling I'll need to implement that new functionality. I am leaning towards Registry.supportsDatasetIdGenerationMode(mode: DatasetIdGenEnum) -> bool rather that something that is explicitly about integer vs UUID.
          Hide
          tjenness Tim Jenness added a comment -

          I like the approach of asking if a dataset ID generation mode is supported. The transfer datasets tests could be cleaned up with this as well.

          Show
          tjenness Tim Jenness added a comment - I like the approach of asking if a dataset ID generation mode is supported. The transfer datasets tests could be cleaned up with this as well.

            People

            Assignee:
            tjenness Tim Jenness
            Reporter:
            tjenness Tim Jenness
            Watchers:
            Jim Bosch, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                CI Builds

                No builds found.