Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-377

Mapper configuration of write options using recipes

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      DM-11332 introduces FITS compression, which may be configured with a variety of options (e.g., compression algorithm, tile size, quantization). We need to be able to select the FITS compression options according to the dataset because compression options relevant/optimal for calexps may not match the compression options relevant/optimal for coadds. The solution proposed in DM-11332 is the introduction of "recipes" for writing datasets.

      A recipe is a set of configuration settings with a symbolic name that may be used to refer to the set. Recipes are defined in obs_base/policy/writeRecipes.yaml; obs packages may define their own custom recipes in obs_whatever/policy/writeRecipes.yaml, which overrides those defined in obs_base. Recipes may be applied to a particular dataset through the mapper policy file (typically obs_whatever/policy/WhateverMapper.yaml) by providing a recipe element for the dataset. The default recipe (used if no recipe is explicitly provided) is called default. Recipes are currently only used for the FitsStorage storage type (used for FITS images), but there's no reason why they couldn't be used for other storage types if required to configure how the writing is done.

      Example recipes

      Here is the proposed obs_base/policy/writeRecipes.yaml:

          # Lossless compression
          default:
            image: &default
              compression:
                scheme: GZIP_SHUFFLE
            mask: *default
            variance: *default
       
          # Basic lossy (quantizing) compression
          basicCompression:
            image: &basicCompression
              compression:
                scheme: RICE
              scaling:
                scheme: STDEV_POSITIVE
                maskPlanes: ["NO_DATA"]
                bitpix: 32
                quantizeLevel: 10.0
                quantizePad: 10.0
            mask: *default
            variance: *basicCompression
      

      Example selection of recipe

      Here's an example in obs_subaru/policy/HscMapper.yaml:

          calexp:
              template: '%(pointing)05d/%(filter)s/corr/CORR-%(visit)07d-%(ccd)03d.fits'
              recipe: basicCompression
      

      Design history

      This design is based on a similar design used in the Pan-STARRS Image Processing Pipeline. In that case, we defined "fits types" which could be selected by their symbolic name in the "file rules". We found that this design worked well, allowing simple selection of the parameters required for writing images. The Pan-STARRS IPP design treated image, mask and variance planes separately, but since in LSST these are treated together, they are together in the recipe.

      Rejected design

      One possibility is to leave off recipes and fold the options into the storage type, maybe something like:

          calexp:
              storage: FitsStorage(image.compression.scheme=GZIP_SHUFFLE, mask.compression.scheme=GZIP_SHUFFLE, variance.compression.scheme=GZIP_SHUFFLE)
              ...
       
          postISRCCD:
              storage: FitsStorage(image.compression.scheme=GZIP_SHUFFLE, mask.compression.scheme=GZIP_SHUFFLE, variance.compression.scheme=GZIP_SHUFFLE)
              ...
      

      It's clear that, even for this simple example, the options are rather long and would have to be repeated several times for each dataset. The use of recipes as proposed simplifies the description and removes the need for repetition.

      More details

      For more information, see:

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Watchers:
                John Swinbank, Kian-Tat Lim, Nate Pease, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel