Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11819

Lossy Compression Working Group


    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Templates:
    • Team:
      Data Facility


      This issue establishes a DM working group to evaluate and recommend options for lossy compression algorithms that compress LSST images in a way that continues to satisfy LSST science use cases (with dark energy being the primary one). All major types of images shall be considered as candidates for compression, including raw data, Processed Visit Images, co-adds, and templates.

      The working group is to:

      • define criteria for "science-usable" lossy-compressed processed images across all LSST image types
      • collect compression algorithm candidates (preferring existing in "off-the-shelf” tools/libraries)
      • evaluate their compression ratios (at "science-usable" quality)
      • evaluate constraints on processing that usage of compression may impose (e.g., avoidance of repeated re-compressions)
      • quantify the savings from application of lossy compression, in the context of the LSST Sizing Model (LDM-144)
      • make recommendations on which image types to lossy-compress, the algorithms to apply, and the description of processing constraints these would impose

      Constraints and prior art:
      This group has emerged as the response to RFC-325 that recognized that user experience will be unacceptably impacted by the long latency required to access the LSST data from tape media. Unfortunately, preliminary analysis indicated that retaining all processed images on disk would be too costly and therefore not feasible, unless lossy compression is applied. The same analysis indicated that storing all raw data on disk (w/o lossy compression) is feasible.

      The LSST has traditionally avoided lossy compression for any of its image data products (including the large co-added images as well as templates retained for each data release). Anecdotal experience from DES and other surveys indicates that lossy compression can be applied, without loss of scientific fidelity. If this is the case, the reduced disk space needs may enable us to retain on low-latency media more data that we otherwise would (rather than regenerate or pull from tape. This group has been convened to study the problem and report on the results.

      The working group should rely as much as possible on prior art found in the literature, and prefer applications of off-the-shelf solution rather than developing custom LSST-specific compression tools.

      The deliverable of this group will be a technical report recommending a scientifically acceptable lossy compression strategy, with a quantification of its impact on the sizing model (see the list under “Charter” for details).

      This group shall complete its work by October 31th, 2017, with fortnightly status updates to the LSST DM Subsystem Scientist and Manager.

      Membership (tentative, except for the chair):

      If you (named above) cannot participate, let Robert Gruendl know. If you're interested in participating in this WG, please notify Robert Gruendl as well. For any questions about the WG, post a comment here.


          Issue Links



              • Assignee:
                gruendl Robert Gruendl
                mjuric Mario Juric
                Ben Emmons [X] (Inactive), Gregory Dubois-Felsmann, John Parejko, John Swinbank, Leanne Guy, Mario Juric, Paul Price, Pim Schellart [X] (Inactive), Robert Gruendl, Tim Jenness, Wil O'Mullane, Zeljko Ivezic
              • Votes:
                0 Vote for this issue
                12 Start watching this issue


                • Due:

                  Summary Panel