Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4171

Butler: change configuration from .paf to something else

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Story Points:
      12
    • Sprint:
      DB_W16_11, DB_W16_12
    • Team:
      Data Access and Database

      Description

      for "something else" there are 2 major options: pex_config (dislike because the 'data' gets executed), and yaml (very well supported, is just data). 3rd option: we could use the sqlite registry, and define a schema for that.

      (sqlite benefits: has support for write locks that will be necessary very soon. Except, need write-once-compare-same mechanism; so we'll need that 1. for normal files and 2. across n nodes. so maybe there's no benefit).

      also, while we're there:
      as noted in DM-4170, registries.py needs to not use astropy and instead should use pyfits. If it's quick, that change should be made here (or turned into a separate ticket)

        Attachments

          Issue Links

            Activity

            No builds found.
            npease Nate Pease [X] (Inactive) created issue -
            jbecla Jacek Becla made changes -
            Field Original Value New Value
            Component/s butler [ 12317 ]
            Team Data Access and Database [ 10204 ]
            rowen Russell Owen made changes -
            Link This issue relates to DM-4170 [ DM-4170 ]
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-2404 [ 16718 ]
            npease Nate Pease [X] (Inactive) made changes -
            Sprint DB_W16_11 [ 171 ]
            Story Points 8
            npease Nate Pease [X] (Inactive) made changes -
            Assignee Nate Pease [ npease ]
            Status To Do [ 10001 ] In Progress [ 3 ]
            npease Nate Pease [X] (Inactive) made changes -
            Description for "something else" there are 2 major options: pex_config (dislike because the 'data' gets executed), and yaml (very well supported, is just data). 3rd option: we could use the sqlite registry, and define a schema for that.

            (sqlite benefits: has support for write locks that will be necessary very soon. Except, need write-once-compare-same mechanism; so we'll need that 1. for normal files and 2. across n nodes. so maybe there's no benefit).
            for "something else" there are 2 major options: pex_config (dislike because the 'data' gets executed), and yaml (very well supported, is just data). 3rd option: we could use the sqlite registry, and define a schema for that.

            (sqlite benefits: has support for write locks that will be necessary very soon. Except, need write-once-compare-same mechanism; so we'll need that 1. for normal files and 2. across n nodes. so maybe there's no benefit).

            also, while we're there:
            as noted in DM-4170, registries.py needs to not use astropy and instead should use pyfits. If it's quick, that change should be made here (or turned into a separate ticket)
            Hide
            tjenness Tim Jenness added a comment -

            Did we decide on YAML in the end?

            Show
            tjenness Tim Jenness added a comment - Did we decide on YAML in the end?
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            yes, YAML.

            Show
            npease Nate Pease [X] (Inactive) added a comment - yes, YAML.
            npease Nate Pease [X] (Inactive) made changes -
            Reviewers Kian-Tat Lim [ ktl ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            npease Nate Pease [X] (Inactive) added a comment - - edited

            I'm also adding code to standardize the sqlite and posix registry api as part of this ticket (I'm combining it because review is going to take some time due to the holiday season and it's easier to add a commit to the review than to set up and test a separate ticket that depends on the changes for this ticket being committed).

            Changes are in daf_butlerUtils and are part of the pull request now.

            Show
            npease Nate Pease [X] (Inactive) added a comment - - edited I'm also adding code to standardize the sqlite and posix registry api as part of this ticket (I'm combining it because review is going to take some time due to the holiday season and it's easier to add a commit to the review than to set up and test a separate ticket that depends on the changes for this ticket being committed). Changes are in daf_butlerUtils and are part of the pull request now.
            jbecla Jacek Becla made changes -
            Sprint DB_W16_11 [ 171 ] DB_W16_11, DB_W16_12 [ 171, 172 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            Hide
            ktl Kian-Tat Lim added a comment -

            Still looking at daf_butlerUtils, but daf_persistence and daf_butler are OK given the comments in the PRs.

            Show
            ktl Kian-Tat Lim added a comment - Still looking at daf_butlerUtils, but daf_persistence and daf_butler are OK given the comments in the PRs.
            npease Nate Pease [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            jbecla Jacek Becla made changes -
            Story Points 8 12
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-4693 [ DM-4693 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering RFC-125 [ RFC-125 ]

              People

              Assignee:
              npease Nate Pease [X] (Inactive)
              Reporter:
              npease Nate Pease [X] (Inactive)
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Kian-Tat Lim, Nate Pease [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.