Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12667

Implement minimal Butler POSIX Datastore prototype

    XMLWordPrintable

Details

    • 8
    • DRP F17-6
    • Data Release Production

    Description

      Implement a minimal Butler Datastore (as defined in DMTN-056) backed by a POSIX filesystem.

      The intent of this ticket is to build a minimum prototype for testing purposes to be used with either the Registry from DM-12613 (or DM-12371).
      This prototype is to be as simple as possible but should be useable to inform decisions about pluggability, chaining, caching and API layering for later versions.

      Attachments

        Issue Links

          Activity

            No builds found.
            pschella Pim Schellart [X] (Inactive) created issue -
            pschella Pim Schellart [X] (Inactive) made changes -
            Field Original Value New Value
            Link This issue relates to DM-12613 [ DM-12613 ]
            pschella Pim Schellart [X] (Inactive) made changes -
            Link This issue relates to DM-12371 [ DM-12371 ]
            pschella Pim Schellart [X] (Inactive) made changes -
            Link This issue relates to DM-12668 [ DM-12668 ]
            jbosch Jim Bosch added a comment -

            A good way to make this minimal and yet functional might be to only have it support a single StorageClass (or a very limited set of StorageClasses), perhaps limited to some with no slicing or composites. Adding support for more complex StorageClasses could then be separate tickets.

            jbosch Jim Bosch added a comment - A good way to make this minimal and yet functional might be to only have it support a single StorageClass (or a very limited set of StorageClasses), perhaps limited to some with no slicing or composites. Adding support for more complex StorageClasses could then be separate tickets.

            Yes, that sounds like a good plan.

            pschella Pim Schellart [X] (Inactive) added a comment - Yes, that sounds like a good plan.
            swinbank John Swinbank made changes -
            Epic Link DM-10586 [ 32520 ]
            pschella Pim Schellart [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]

            Ready for initial review of the general structure. There is virtually unlimited scope for improvement and cleanup/robustification, but the basic structure is there and all required methods to support a prototype Butler are implemented.

            Please be brutal in this review, but keep in mind that this is a prototype implementation meant to verify the design and not (yet) production code.

            Things that should still be done, but probably on separate small tickets are:

            • Improve configuration
            • Investigate adding a buffering layer to Formatter (write to buffer -> write buffer to file)
            • Add more default storage classes
            • Add basic building blocks for chaining (caching, multiple output formats for a single put, etc.) (mostly trivial separate implementations of the Datastore interface that just forward calls based on some further configuration)
            • Add support for composites
            pschella Pim Schellart [X] (Inactive) added a comment - Ready for initial review of the general structure. There is virtually unlimited scope for improvement and cleanup/robustification, but the basic structure is there and all required methods to support a prototype Butler are implemented. Please be brutal in this review, but keep in mind that this is a prototype implementation meant to verify the design and not (yet) production code. Things that should still be done, but probably on separate small tickets are: Improve configuration Investigate adding a buffering layer to Formatter (write to buffer -> write buffer to file) Add more default storage classes Add basic building blocks for chaining (caching, multiple output formats for a single put , etc.) (mostly trivial separate implementations of the Datastore interface that just forward calls based on some further configuration) Add support for composites
            pschella Pim Schellart [X] (Inactive) made changes -
            Reviewers Simon Krughoff [ krughoff ]
            Status In Progress [ 3 ] In Review [ 10004 ]

            To clarify, there are three levels of plugability/reuseability in the current design:

            • Different implementations of the Datastore interface support different storage types (POSIX/HTTP/Object Store), these can then reuse:
            • Different implementations of the Formatter interface that read and write files of a particular StorageClass (which can be configured and swapped out dynamically based on a config file).
            • Different Datastore instances (of different type) can be chained to support things like a Proxy, storing to multiple file formats, etc.
            pschella Pim Schellart [X] (Inactive) added a comment - To clarify, there are three levels of plugability/reuseability in the current design: Different implementations of the Datastore interface support different storage types (POSIX/HTTP/Object Store), these can then reuse: Different implementations of the Formatter interface that read and write files of a particular StorageClass (which can be configured and swapped out dynamically based on a config file). Different Datastore instances (of different type) can be chained to support things like a Proxy, storing to multiple file formats, etc.
            yusra Yusra AlSayyad made changes -
            Sprint DRP F17-6 [ 625 ] DRP F17-6, DRP S18-1 [ 625, 668 ]
            yusra Yusra AlSayyad made changes -
            Rank Ranked higher

            Reviewed. I hope it's helpful.

            krughoff Simon Krughoff (Inactive) added a comment - Reviewed. I hope it's helpful.
            krughoff Simon Krughoff (Inactive) made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]

            Thanks for the review!

            pschella Pim Schellart [X] (Inactive) added a comment - Thanks for the review!
            pschella Pim Schellart [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            pschella Pim Schellart [X] (Inactive) made changes -
            Labels jarvis gen3-middleware
            fritzm Fritz Mueller made changes -
            Sprint DRP F17-6, DRP S18-1 [ 625, 668 ] DRP F17-6 [ 625 ]
            tjenness Tim Jenness made changes -
            Component/s daf_butler [ 11300 ]

            People

              pschella Pim Schellart [X] (Inactive)
              pschella Pim Schellart [X] (Inactive)
              Simon Krughoff (Inactive)
              Jim Bosch, John Swinbank, Pim Schellart [X] (Inactive), Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.