Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21246

Allow deferred passing of run/collection to Butler

    Details

    • Story Points:
      2
    • Team:
      Architecture

      Description

      While working on ingest and especially gen2convert, I've found it frustrating to need to pass a collection/run to Butler at construction, rather than later, when calling get/put/ingest.  We often need an object that represents the combination of a Registry and a Datastore without specifying a collection/run.

      I think we should add optional "run" kwargs to all Butler write operations and "collection" kwargs to all read operations, and make the corresponding construction arguments an optional way to provide defaults for these.

      As a side note, I think I'd like to eventually move to having Butler support lookup with ordered sequence of collections rather than just a single one.  We could anticipate that by making the new kwargs for read operations a "collections" sequence rather than than a "collection" scalar, even if we don't implement support for more than one yet, but I'm also willing to defer that for later.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I'm fine with run not being in the config. It seems odd that it's there at all given that multiple collections/runs are the point of a butler.

            Show
            tjenness Tim Jenness added a comment - I'm fine with run not being in the config. It seems odd that it's there at all given that multiple collections/runs are the point of a butler.
            Hide
            jbosch Jim Bosch added a comment -

            Fixes for DM-21370, DM-23180, DM-19826 are included on the branch for this ticket.

            It looks like deferring work on DM-19617 and DM-16876 until DM-21849 or later makes more sense than doing them here.

            Show
            jbosch Jim Bosch added a comment - Fixes for DM-21370 , DM-23180 , DM-19826 are included on the branch for this ticket. It looks like deferring work on DM-19617 and DM-16876 until DM-21849 or later makes more sense than doing them here.
            Hide
            jbosch Jim Bosch added a comment -

            Tim Jenness, another review for you, I'm afraid. This is mostly cleanups and minor refactors in butler itself (some of which turned out to have tickets of their own) until it gets the point of this ticket in the last few commits.

            I've deliberately not gone into the closely-related tickets that are impacted by RFC-663 (I actually tried and realized it was a tangled knot that merited an RFC, then put that in-progress commit on a different branch).

            Show
            jbosch Jim Bosch added a comment - Tim Jenness , another review for you, I'm afraid. This is mostly cleanups and minor refactors in butler itself (some of which turned out to have tickets of their own) until it gets the point of this ticket in the last few commits. I've deliberately not gone into the closely-related tickets that are impacted by RFC-663 (I actually tried and realized it was a tangled knot that merited an RFC, then put that in-progress commit on a different branch).
            Hide
            tjenness Tim Jenness added a comment -

            Looks good. Minor coments on PR.

            Show
            tjenness Tim Jenness added a comment - Looks good. Minor coments on PR.
            Hide
            jbosch Jim Bosch added a comment -

            I've added the missing test to daf_butler, but also removed some usage of that functionality in obs_base. Mind taking a look at that (small) PR?

            Show
            jbosch Jim Bosch added a comment - I've added the missing test to daf_butler, but also removed some usage of that functionality in obs_base. Mind taking a look at that (small) PR ?

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Tim Jenness
                Watchers:
                Christopher Waters, Gregory Dubois-Felsmann, Hsin-Fang Chiang, Jim Bosch, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel