Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:
-
Story Points:4
-
Epic Link:
-
Sprint:DRP F17-6
-
Team:Data Release Production
Description
Use the prototype Registry from either DM-12613 or DM-12371 and the prototype Datastore from DM-12667 to implement a prototype Butler (as described in DMTN-056).
The only supported operations are Butler::get, Butler::put and Butler::unlink with a DatasetLabel (e.g. no querying, data-graphs or transfers).
The intent of this prototype is to inform design decisions for the final version. It should not be throw-away code but is allowed to be rough around the edges.
Attachments
Issue Links
Activity
I've read through the pull request and they look reasonable. I didn't look at other parts of the butler prototype.
Still quite unfamiliar with the Gen 3 Butler but I'm slowly getting the ideas. A few questions come to mind:
- For a read-only Butler, in what use cases would unlink and markInputUsed be needed? Or are they not inteneded for use in the read-only Butler?
- Can a Butler be given multiple CollectionTags via its config? I thought yes but it doesn't seem included in the current design/codes yet? Or would they be taken care of somewhere else?
- Why do we need both DatasetLabel and DatasetRef?
Hsin-Fang Chiang
added a comment - I've read through the pull request and they look reasonable. I didn't look at other parts of the butler prototype.
Still quite unfamiliar with the Gen 3 Butler but I'm slowly getting the ideas. A few questions come to mind:
For a read-only Butler, in what use cases would unlink and markInputUsed be needed? Or are they not inteneded for use in the read-only Butler?
Can a Butler be given multiple CollectionTags via its config? I thought yes but it doesn't seem included in the current design/codes yet? Or would they be taken care of somewhere else?
Why do we need both DatasetLabel and DatasetRef ?
Thanks for the review! Comments are noted and taken into account. There will be some way to support multiple collection tags, but none exists as of yet. Read-only butlers should indeed not implement those methods.
Pim Schellart [X] (Inactive)
added a comment - Thanks for the review! Comments are noted and taken into account. There will be some way to support multiple collection tags, but none exists as of yet. Read-only butlers should indeed not implement those methods. `DatasetLabel` can be created by a user/tool without needing to be bound to a registry. In contrast a `DatasetRef` is always bound. Added doctrings to `Registry` to clarify.
Pim Schellart [X] (Inactive)
added a comment - `DatasetLabel` can be created by a user/tool without needing to be bound to a registry. In contrast a `DatasetRef` is always bound. Added doctrings to `Registry` to clarify. I understand (or misunderstand) that a registry is always needed for a Butler, while this registry can be sqlite, Oracle, or any database. In what cases would there be butler operations without a registry? May you say more about what you mean by "without needing to be bound to a registry" or examples how DatasetLabel and DatasetRef are used?
Hsin-Fang Chiang
added a comment - I understand (or misunderstand) that a registry is always needed for a Butler, while this registry can be sqlite, Oracle, or any database. In what cases would there be butler operations without a registry? May you say more about what you mean by "without needing to be bound to a registry" or examples how DatasetLabel and DatasetRef are used? 
I assigned both of you as reviewers for this small ticket (feel free to object), since you are already reviewing its constituent components, and this gives a good idea about how they are linked together and used in practice.
As before, please note that this is by no means robust production code yet. So please mainly direct your review comments towards the overall design.
The main other thing to ignore in this for now is configuration, which I intend to cleanup on the next ticket in this series (the intent being that the particular Registry and Datastore to be instantiated are to be defined (along with their settings) in a YAML configuration file (the user will then just have to provide this file and not the actual instances)). But that is for a future ticket.