Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23609

Move dataset components out of registry and back to datastore

    Details

    • Templates:
    • Story Points:
      15
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Jim Bosch reports that none of the pipeline tasks being developed ever need to work with specific components at the registry level. This leads us to think that registry does not need to know about components and does not need to list every single component of each dataset in the registry.

      We therefore want to go back to the original design (and early implementation) where registry only knows about the single dataset and datastore knows about components. Datastore configuration controls whether assembly/disassembly should happen for a given dataset (and different datastores could do their own thing). This will mean that when disassembly does not occur, significant savings in row count in the registry will be made.

      To support composite datasets where a wcs can come from one place and pixel data from another, we need to implement the concept of a virtual dataset (with a new API) where registry can tell datastore that a new dataset exists that is a combination of components from different datasets already known to datastore.

      Not seen as a blocker for Gen2, but will lead to some additional registry instability (which may require us to support both registry concepts together).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Jim Bosch, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Summary Panel