Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13876

Write a ParquetStorage Butler storage type

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Story Points:
      10
    • Team:
      Data Release Production

      Description

      DM-13770 implemented butler-enabled reading/writing of Parquet tables via a FitsCatalogStorage shim.  However, in order to take advantage of the column-store nature, it is really necessary to define a new storage type that will enable extraction of specified columns without loading in the whole table.  Jim Bosch suggests that this "basically boils down to grepping for all the places e.g. FitsStorage appears in obs_base and daf_persistence, and adding new clauses to various if blocks for the new storage type."  Additionally, a model for adding extra keywords to be passed to the loading function can be found with FitsStorage, which allows, e.g. butler.get("calexp_sub", dataId=..., bbox=...).

        Attachments

          Activity

            People

            • Assignee:
              tmorton Tim Morton
              Reporter:
              tmorton Tim Morton
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Jim Bosch, John Swinbank, Kian-Tat Lim, Tim Morton, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel