Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13876

Write a ParquetStorage Butler storage type

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Story Points:
      10
    • Team:
      Data Release Production

      Description

      DM-13770 implemented butler-enabled reading/writing of Parquet tables via a FitsCatalogStorage shim.  However, in order to take advantage of the column-store nature, it is really necessary to define a new storage type that will enable extraction of specified columns without loading in the whole table.  Jim Bosch suggests that this "basically boils down to grepping for all the places e.g. FitsStorage appears in obs_base and daf_persistence, and adding new clauses to various if blocks for the new storage type."  Additionally, a model for adding extra keywords to be passed to the loading function can be found with FitsStorage, which allows, e.g. butler.get("calexp_sub", dataId=..., bbox=...).

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                tmorton Tim Morton
                Reporter:
                tmorton Tim Morton
                Reviewers:
                Kian-Tat Lim
                Watchers:
                Jim Bosch, John Swinbank, Kian-Tat Lim, Tim Morton, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel