Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36766

Write migration scripts for adding obscore to USDF repos

    XMLWordPrintable

    Details

      Description

      Once pg_sphere is installed and we agreed on obscore configurations (DM-36701) we need to run migrations to add everyhting to each of the repos.

      My initial idea is that migration script would just add obscore config to butler_attributes and create the obscore teable with a correct schema (which is based on config). For actual filling of the table from pre-existing data I want to implement a separate script which will run later, that way the migration can be short and we don't need a downtime for it. The population script can run concurrently with other users, so we should not need any disruption.

      Just like for dimensions manager we'll have two separate migrations branches, one for the manager class itself, and another for configuration. It is also likely that we'll need separate migration branch for each repo, as each of them will heave an independent configuration. IT makes sense to start with a single repo as a demonstrator to show people how it's going to look like. It is also less critical for obscore to do everything right on the first iteration, we can always delete and re-create obscore table again.

        Attachments

          Activity

          No builds found.
          salnikov Andy Salnikov created issue -
          salnikov Andy Salnikov made changes -
          Field Original Value New Value
          Epic Link DM-30629 [ 513192 ] PREOPS-1593 [ 2393150 ]
          salnikov Andy Salnikov made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          salnikov Andy Salnikov added a comment -

          Tim Jenness, this should be ready for review, we are still waiting for pgsphere so there is no rush. There are three packages involved:

          • daf_butler - a bunch of small improvements to support new migration scripts
          • daf_butler_migrate - actual migration scripts, plus some changes to supporting classes
          • dax_obscore - new CLI command to fill obscore table with pre-existing data after schema migration

          Jenkins is running but it only covers daf_butler, I've done testing for the rest on my local copy of OGA repo.

          Show
          salnikov Andy Salnikov added a comment - Tim Jenness , this should be ready for review, we are still waiting for pgsphere so there is no rush. There are three packages involved: daf_butler - a bunch of small improvements to support new migration scripts daf_butler_migrate - actual migration scripts, plus some changes to supporting classes dax_obscore - new CLI command to fill obscore table with pre-existing data after schema migration Jenkins is running but it only covers daf_butler, I've done testing for the rest on my local copy of OGA repo.
          salnikov Andy Salnikov made changes -
          Reviewers Tim Jenness [ tjenness ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          salnikov Andy Salnikov added a comment -

          Tim Jenness, one general comment - dax_obscore now has three commands that it adds to butler CLI - obscore-export, obscore-set-exposure-regions, obscore-update-table. I probably makes sense to transform all of them into one sub-command with three sub-sub-commands (e.g. butler obscore export). Let me know if you want me to do it on this ticket.

          Show
          salnikov Andy Salnikov added a comment - Tim Jenness , one general comment - dax_obscore now has three commands that it adds to butler CLI - obscore-export, obscore-set-exposure-regions, obscore-update-table. I probably makes sense to transform all of them into one sub-command with three sub-sub-commands (e.g. butler obscore export ). Let me know if you want me to do it on this ticket.
          Hide
          tjenness Tim Jenness added a comment -

          My first reaction is butler obscore is the right answer.

          Show
          tjenness Tim Jenness added a comment - My first reaction is butler obscore is the right answer.
          Hide
          salnikov Andy Salnikov added a comment -

          OK, I'll change that. I don't think we need a deprecation period, there should not be many users of those commands out there.

          Show
          salnikov Andy Salnikov added a comment - OK, I'll change that. I don't think we need a deprecation period, there should not be many users of those commands out there.
          Hide
          salnikov Andy Salnikov added a comment -

          I added one more commit which makes "obscore" a separate command group.

          Show
          salnikov Andy Salnikov added a comment - I added one more commit which makes "obscore" a separate command group.
          Hide
          tjenness Tim Jenness added a comment -

          Sorry it took me so long to get to it. Looks okay.

          Show
          tjenness Tim Jenness added a comment - Sorry it took me so long to get to it. Looks okay.
          tjenness Tim Jenness made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          salnikov Andy Salnikov made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]

            People

            Assignee:
            salnikov Andy Salnikov
            Reporter:
            salnikov Andy Salnikov
            Reviewers:
            Tim Jenness
            Watchers:
            Andy Salnikov, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.