Fix Version/s: None
This issue establishes a DM working group to evaluate and recommend options for lossy compression algorithms that compress LSST images in a way that continues to satisfy LSST science use cases (with dark energy being the primary one). All major types of images shall be considered as candidates for compression, including raw data, Processed Visit Images, co-adds, and templates.
The working group is to:
- define criteria for "science-usable" lossy-compressed processed images across all LSST image types
- collect compression algorithm candidates (preferring existing in "off-the-shelf” tools/libraries)
- evaluate their compression ratios (at "science-usable" quality)
- evaluate constraints on processing that usage of compression may impose (e.g., avoidance of repeated re-compressions)
- quantify the savings from application of lossy compression, in the context of the LSST Sizing Model (LDM-144)
- make recommendations on which image types to lossy-compress, the algorithms to apply, and the description of processing constraints these would impose
Constraints and prior art:
This group has emerged as the response to RFC-325 that recognized that user experience will be unacceptably impacted by the long latency required to access the LSST data from tape media. Unfortunately, preliminary analysis indicated that retaining all processed images on disk would be too costly and therefore not feasible, unless lossy compression is applied. The same analysis indicated that storing all raw data on disk (w/o lossy compression) is feasible.
The LSST has traditionally avoided lossy compression for any of its image data products (including the large co-added images as well as templates retained for each data release). Anecdotal experience from DES and other surveys indicates that lossy compression can be applied, without loss of scientific fidelity. If this is the case, the reduced disk space needs may enable us to retain on low-latency media more data that we otherwise would (rather than regenerate or pull from tape. This group has been convened to study the problem and report on the results.
The working group should rely as much as possible on prior art found in the literature, and prefer applications of off-the-shelf solution rather than developing custom LSST-specific compression tools.
The deliverable of this group will be a technical report recommending a scientifically acceptable lossy compression strategy, with a quantification of its impact on the sizing model (see the list under “Charter” for details).
This group shall complete its work by October 31th, 2017, with fortnightly status updates to the LSST DM Subsystem Scientist and Manager.
Membership (tentative, except for the chair):
- Robert Gruendl (Chair)
- Eric Morganson (NCSA; analyzed the problem for DES)
- Bob Armstrong (weak lensing concerns)
- John Parejko (image differencing concerns)
- Paul Price (general experience in this area, understanding of pipeline architecture)
If you (named above) cannot participate, let Robert Gruendl know. If you're interested in participating in this WG, please notify Robert Gruendl as well. For any questions about the WG, post a comment here.