Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26407

Change how dimensions are stored in a Butler repository

    XMLWordPrintable

    Details

      Description

      Dimensions.yaml should be locked in and read-only as soon as the schema is created during the makeRepo phase. At the moment there is nothing stopping a repository from breaking if dimensions.yaml is updated in daf_butler and there is nothing stopping a user editing the config file after repository creation and breaking things.

      We propose to remove dimensions.yaml from ButlerConfig and have it as an explicit argument for makeRepo (defaulting to pkg://lsst.daf.butler/configs/dimensions.yaml) which will then ensure that the dimensions configuration is locked in for that repository.

      We will still require that it be possible to migrate a registry to a new dimensions definition but that migration should be by explicit request.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27033 Integration of pre-middleware-release dimensions changes

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27266 Switch serialized Dimensions config to JSON inside database

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27256 Add JSON support to butler Config

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            32 older comments
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Is JSON better than YAML? Are you worried about line breaks? With YAML you can at least dump that into dimension.yaml without going through Config class.

            Show
            salnikov Andy Salnikov added a comment - Is JSON better than YAML? Are you worried about line breaks? With YAML you can at least dump that into dimension.yaml without going through Config class.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            It's more a general comment about storing all the \n and whitespace characters in the table when that's exactly what JSON is designed to solve.

            Supporting JSON natively in Config is not difficult. Kian-Tat Lim do you have an opinion on storing JSON vs YAML in database?

            Show
            tjenness Tim Jenness added a comment - It's more a general comment about storing all the \n and whitespace characters in the table when that's exactly what JSON is designed to solve. Supporting JSON natively in Config is not difficult. Kian-Tat Lim do you have an opinion on storing JSON vs YAML in database?
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            My general philosophy is: "YAML is better if humans are writing/reading it. JSON is better if machines are writing/reading it."

            Show
            ktl Kian-Tat Lim added a comment - My general philosophy is: "YAML is better if humans are writing/reading it. JSON is better if machines are writing/reading it."
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            To make everyone (un)happy we should do XML then 

            Show
            salnikov Andy Salnikov added a comment - To make everyone (un)happy we should do XML then 
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            JSON form of dimensions.yaml is 10% smaller than YAML version. I think Kian-Tat Lim is on the side of switching to JSON. I will do that once DM-27256 is merged (which adds JSON support to Config).

            Show
            tjenness Tim Jenness added a comment - JSON form of dimensions.yaml is 10% smaller than YAML version. I think Kian-Tat Lim is on the side of switching to JSON. I will do that once DM-27256 is merged (which adds JSON support to Config).

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Jim Bosch
              Watchers:
              Andy Salnikov, Jim Bosch, Kian-Tat Lim, Michelle Gower, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.