Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24288

Prototype datastore changes to support dataset disassembly/assembly

    Details

    • Templates:
    • Story Points:
      6
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      After some discussion we have decided that it might be better for datastore to be in charge of assembly/disassembly of datasets. Currently registry creates 7 rows per raw with almost none of those components being relevant. We did it this way to allow pipelines to specify components directly. We very rarely want to do this though and it would be more efficient in database storage to have a single row for a dataset in registry and move [dis]assembly to the datastore layer. Then we can instead write a single row if we do not disassemble and multiple rows if we do disassemble (and no row at all for components that were not present). This is similar to the original implementation of datastore.

      This ticket is to check that the approach is feasible and also whether it can be implemented independently of registry to allow us to experiment with both approaches in parallel.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel