Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35069

Devise a scheme for butler export YAML migration

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: daf_butler
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      It's becoming increasingly clear that supporting schema migrations of butler registries is only part of the migration work that is necessary.

      The missing part is migrating all the export.yaml files that are being used to the current schema. In DM-33942 I modified import to understand the old and new layouts of visit definitions but that is only part of the story.

      For registry migrations we add additional logic to fill in missing record metadata (and allow command lines to specify some values) but butler import can not do any of this. This means that we end up in situations where importing a yaml drops the new metadata fields but migrating a registry fills in gaps. This happened today with one of the AP packages that uses export yaml for exposure records and then ingests raws (which tries to recreate the exposure records). Telling them to force overwrite of the records might be an option (but there is no command line) but I think it's warning us of a bigger issue.

      We need to work out a way to be able to migrate export yaml files. We now store the universe version/namespace in files as well as the export format itself so this is tractable. Then we could make it so that butler import only has to support the newest variant and people can run butler migrate on their YAML files to bring them up to date.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            Just found this issue. For the record, I think registry migration would have solved the problem with AP (though DM-35071 will be taking the "force overwrite" route, since I think we do want to be backward-compatible with older packages).

            Show
            krzys Krzysztof Findeisen added a comment - Just found this issue. For the record, I think registry migration would have solved the problem with AP (though DM-35071 will be taking the "force overwrite" route, since I think we do want to be backward-compatible with older packages).

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Andy Salnikov, Jim Bosch, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.