Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12667

Implement minimal Butler POSIX Datastore prototype

    XMLWordPrintable

Details

    • 8
    • DRP F17-6
    • Data Release Production

    Description

      Implement a minimal Butler Datastore (as defined in DMTN-056) backed by a POSIX filesystem.

      The intent of this ticket is to build a minimum prototype for testing purposes to be used with either the Registry from DM-12613 (or DM-12371).
      This prototype is to be as simple as possible but should be useable to inform decisions about pluggability, chaining, caching and API layering for later versions.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            A good way to make this minimal and yet functional might be to only have it support a single StorageClass (or a very limited set of StorageClasses), perhaps limited to some with no slicing or composites. Adding support for more complex StorageClasses could then be separate tickets.

            jbosch Jim Bosch added a comment - A good way to make this minimal and yet functional might be to only have it support a single StorageClass (or a very limited set of StorageClasses), perhaps limited to some with no slicing or composites. Adding support for more complex StorageClasses could then be separate tickets.

            Yes, that sounds like a good plan.

            pschella Pim Schellart [X] (Inactive) added a comment - Yes, that sounds like a good plan.

            Ready for initial review of the general structure. There is virtually unlimited scope for improvement and cleanup/robustification, but the basic structure is there and all required methods to support a prototype Butler are implemented.

            Please be brutal in this review, but keep in mind that this is a prototype implementation meant to verify the design and not (yet) production code.

            Things that should still be done, but probably on separate small tickets are:

            • Improve configuration
            • Investigate adding a buffering layer to Formatter (write to buffer -> write buffer to file)
            • Add more default storage classes
            • Add basic building blocks for chaining (caching, multiple output formats for a single put, etc.) (mostly trivial separate implementations of the Datastore interface that just forward calls based on some further configuration)
            • Add support for composites
            pschella Pim Schellart [X] (Inactive) added a comment - Ready for initial review of the general structure. There is virtually unlimited scope for improvement and cleanup/robustification, but the basic structure is there and all required methods to support a prototype Butler are implemented. Please be brutal in this review, but keep in mind that this is a prototype implementation meant to verify the design and not (yet) production code. Things that should still be done, but probably on separate small tickets are: Improve configuration Investigate adding a buffering layer to Formatter (write to buffer -> write buffer to file) Add more default storage classes Add basic building blocks for chaining (caching, multiple output formats for a single put , etc.) (mostly trivial separate implementations of the Datastore interface that just forward calls based on some further configuration) Add support for composites

            To clarify, there are three levels of plugability/reuseability in the current design:

            • Different implementations of the Datastore interface support different storage types (POSIX/HTTP/Object Store), these can then reuse:
            • Different implementations of the Formatter interface that read and write files of a particular StorageClass (which can be configured and swapped out dynamically based on a config file).
            • Different Datastore instances (of different type) can be chained to support things like a Proxy, storing to multiple file formats, etc.
            pschella Pim Schellart [X] (Inactive) added a comment - To clarify, there are three levels of plugability/reuseability in the current design: Different implementations of the Datastore interface support different storage types (POSIX/HTTP/Object Store), these can then reuse: Different implementations of the Formatter interface that read and write files of a particular StorageClass (which can be configured and swapped out dynamically based on a config file). Different Datastore instances (of different type) can be chained to support things like a Proxy, storing to multiple file formats, etc.

            Reviewed. I hope it's helpful.

            krughoff Simon Krughoff (Inactive) added a comment - Reviewed. I hope it's helpful.

            Thanks for the review!

            pschella Pim Schellart [X] (Inactive) added a comment - Thanks for the review!

            People

              pschella Pim Schellart [X] (Inactive)
              pschella Pim Schellart [X] (Inactive)
              Simon Krughoff (Inactive)
              Jim Bosch, John Swinbank, Pim Schellart [X] (Inactive), Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.