Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4171

Butler: change configuration from .paf to something else

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Story Points:
      12
    • Sprint:
      DB_W16_11, DB_W16_12
    • Team:
      Data Access and Database

      Description

      for "something else" there are 2 major options: pex_config (dislike because the 'data' gets executed), and yaml (very well supported, is just data). 3rd option: we could use the sqlite registry, and define a schema for that.

      (sqlite benefits: has support for write locks that will be necessary very soon. Except, need write-once-compare-same mechanism; so we'll need that 1. for normal files and 2. across n nodes. so maybe there's no benefit).

      also, while we're there:
      as noted in DM-4170, registries.py needs to not use astropy and instead should use pyfits. If it's quick, that change should be made here (or turned into a separate ticket)

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            Still looking at daf_butlerUtils, but daf_persistence and daf_butler are OK given the comments in the PRs.

            Show
            ktl Kian-Tat Lim added a comment - Still looking at daf_butlerUtils, but daf_persistence and daf_butler are OK given the comments in the PRs.
            Hide
            npease Nate Pease [X] (Inactive) added a comment - - edited

            I'm also adding code to standardize the sqlite and posix registry api as part of this ticket (I'm combining it because review is going to take some time due to the holiday season and it's easier to add a commit to the review than to set up and test a separate ticket that depends on the changes for this ticket being committed).

            Changes are in daf_butlerUtils and are part of the pull request now.

            Show
            npease Nate Pease [X] (Inactive) added a comment - - edited I'm also adding code to standardize the sqlite and posix registry api as part of this ticket (I'm combining it because review is going to take some time due to the holiday season and it's easier to add a commit to the review than to set up and test a separate ticket that depends on the changes for this ticket being committed). Changes are in daf_butlerUtils and are part of the pull request now.
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            yes, YAML.

            Show
            npease Nate Pease [X] (Inactive) added a comment - yes, YAML.
            Hide
            tjenness Tim Jenness added a comment -

            Did we decide on YAML in the end?

            Show
            tjenness Tim Jenness added a comment - Did we decide on YAML in the end?

              People

              Assignee:
              npease Nate Pease [X] (Inactive)
              Reporter:
              npease Nate Pease [X] (Inactive)
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Kian-Tat Lim, Nate Pease [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.