Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4924

Please provide transparent access to compressed files

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Team:
      Data Access and Database

      Description

      I just noticed that if we compress files on disk (using gzip in this case) that the butler can't read them, although cfitsio can. That is,
      calexp = butler.get("calexp", dataId)
      fails but
      calexp = afwImage.ExposureF(butler.get("calexp_filename", dataId))
      works.

      SDSS's butler-equivalent checked for a variety of suffixes. for example
      ["", ".gz", ".fz"]
      and handled this transparently, searching for files with one of these suffixes appended until a match was found.

      Please provide this, or equivalent, functionality.

        Attachments

          Issue Links

            Activity

            Hide
            npease Nate Pease added a comment -

            I think this will be a behavior of pluggable object de/serialization. TBD. (discussion currently going on in c.l.o. at https://community.lsst.org/t/how-does-the-butler-support-compression/502

            Show
            npease Nate Pease added a comment - I think this will be a behavior of pluggable object de/serialization. TBD. (discussion currently going on in c.l.o. at https://community.lsst.org/t/how-does-the-butler-support-compression/502
            Hide
            npease Nate Pease added a comment -

            FYI Robert Lupton
            this was done, as tickets/DM-8254

            Show
            npease Nate Pease added a comment - FYI Robert Lupton this was done, as tickets/ DM-8254

              People

              • Assignee:
                Unassigned
                Reporter:
                rhl Robert Lupton
                Watchers:
                Nate Pease, Robert Lupton
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel