Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13353

Prototype implementation of (optionally) compressed Datasets in Datastore

    Details

    • Story Points:
      6
    • Team:
      Architecture

      Description

      Implement a prototype that supports get/put on (optionally) compressed Datasets.

      Needs to propagate / store compression options. Butler can possibly use ChainedDatastore to store both compressed and non-compressed versions.

      Interaction with Registry is a separate ticket.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            I don't know of anyone that has customised compression options, apart from me. I think that means that the vast majority of people don't need a generic implementation. Though I hesitate to suggest that a generic implementation is not a priority (because there's a good chance that that means it would never get done, and I think you'd be crazy not to allow people to tweak formatter parameters), that's not inaccurate.

            Show
            price Paul Price added a comment - I don't know of anyone that has customised compression options, apart from me. I think that means that the vast majority of people don't need a generic implementation. Though I hesitate to suggest that a generic implementation is not a priority (because there's a good chance that that means it would never get done, and I think you'd be crazy not to allow people to tweak formatter parameters), that's not inaccurate.
            Hide
            tjenness Tim Jenness added a comment -

            I've added write parameters in gen3 datastore config files for formatters in general to DM-25178 but that was mostly intending to allow per-recipe selection – I was shying away from allowing a full FITS compression recipe descriptions there and was intending to use your approach of specifying compression options by label.

            I'll have a think about moving writeRecipes.yaml into butler datastore configuration.

            Show
            tjenness Tim Jenness added a comment - I've added write parameters in gen3 datastore config files for formatters in general to DM-25178 but that was mostly intending to allow per-recipe selection – I was shying away from allowing a full FITS compression recipe descriptions there and was intending to use your approach of specifying compression options by label. I'll have a think about moving writeRecipes.yaml into butler datastore configuration.
            Hide
            tjenness Tim Jenness added a comment -

            Paul Price thank you for agreeing to review. I followed your advice and added general support for writeRecipes to daf_butler and then modified FitsExposureFormatter in obs_base to use the recipes. I took your validation code from cameraMapper.

            Show
            tjenness Tim Jenness added a comment - Paul Price thank you for agreeing to review. I followed your advice and added general support for writeRecipes to daf_butler and then modified FitsExposureFormatter in obs_base to use the recipes. I took your validation code from cameraMapper.
            Hide
            tjenness Tim Jenness added a comment -

            Paul Price thanks for looking. Jim Bosch says he will finish the review from a butler perspective.

            Show
            tjenness Tim Jenness added a comment - Paul Price thanks for looking. Jim Bosch says he will finish the review from a butler perspective.
            Hide
            price Paul Price added a comment -

            Very nicely done!

            You may want someone else to look at the butler-specific changes, where I am a bit overwhelmed; but the compression-related changes look good to me.

            Show
            price Paul Price added a comment - Very nicely done! You may want someone else to look at the butler-specific changes, where I am a bit overwhelmed; but the compression-related changes look good to me.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                pschella Pim Schellart [X] (Inactive)
                Reviewers:
                Jim Bosch, Paul Price
                Watchers:
                Jim Bosch, Michelle Gower, Paul Price, Pim Schellart [X] (Inactive), Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel