Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-377

Mapper configuration of write options using recipes


    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:


      DM-11332 introduces FITS compression, which may be configured with a variety of options (e.g., compression algorithm, tile size, quantization). We need to be able to select the FITS compression options according to the dataset because compression options relevant/optimal for calexps may not match the compression options relevant/optimal for coadds. The solution proposed in DM-11332 is the introduction of "recipes" for writing datasets.

      A recipe is a set of configuration settings with a symbolic name that may be used to refer to the set. Recipes are defined in obs_base/policy/writeRecipes.yaml; obs packages may define their own custom recipes in obs_whatever/policy/writeRecipes.yaml, which overrides those defined in obs_base. Recipes may be applied to a particular dataset through the mapper policy file (typically obs_whatever/policy/WhateverMapper.yaml) by providing a recipe element for the dataset. The default recipe (used if no recipe is explicitly provided) is called default. Recipes are currently only used for the FitsStorage storage type (used for FITS images), but there's no reason why they couldn't be used for other storage types if required to configure how the writing is done.

      Example recipes

      Here is the proposed obs_base/policy/writeRecipes.yaml:

          # Lossless compression
            image: &default
                scheme: GZIP_SHUFFLE
            mask: *default
            variance: *default
          # Basic lossy (quantizing) compression
            image: &basicCompression
                scheme: RICE
                scheme: STDEV_POSITIVE
                maskPlanes: ["NO_DATA"]
                bitpix: 32
                quantizeLevel: 10.0
                quantizePad: 10.0
            mask: *default
            variance: *basicCompression

      Example selection of recipe

      Here's an example in obs_subaru/policy/HscMapper.yaml:

              template: '%(pointing)05d/%(filter)s/corr/CORR-%(visit)07d-%(ccd)03d.fits'
              recipe: basicCompression

      Design history

      This design is based on a similar design used in the Pan-STARRS Image Processing Pipeline. In that case, we defined "fits types" which could be selected by their symbolic name in the "file rules". We found that this design worked well, allowing simple selection of the parameters required for writing images. The Pan-STARRS IPP design treated image, mask and variance planes separately, but since in LSST these are treated together, they are together in the recipe.

      Rejected design

      One possibility is to leave off recipes and fold the options into the storage type, maybe something like:

              storage: FitsStorage(image.compression.scheme=GZIP_SHUFFLE, mask.compression.scheme=GZIP_SHUFFLE, variance.compression.scheme=GZIP_SHUFFLE)
              storage: FitsStorage(image.compression.scheme=GZIP_SHUFFLE, mask.compression.scheme=GZIP_SHUFFLE, variance.compression.scheme=GZIP_SHUFFLE)

      It's clear that, even for this simple example, the options are rather long and would have to be repeated several times for each dataset. The use of recipes as proposed simplifies the description and removes the need for repetition.

      More details

      For more information, see:


          Issue Links



              • Assignee:
                price Paul Price
                price Paul Price
                John Swinbank, Kian-Tat Lim, Nate Pease, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                5 Start watching this issue


                • Created:
                  Planned End:

                  Summary Panel