Background

Currently, filter information in exposures is held by an lsst.afw.image.Filter object attached to the exposure. This class provides information such as aliases and wavelength limits to algorithmic code, but has proven inflexible in practice. RFC-624 proposed replacing this class with a hierarchy of Filter classes, but implementation stalled over design and complexity questions.

The following is a new proposal for a new system to represent filters in algorithm code, based on additional feedback we've solicited from various stakeholders. If adopted, this RFC will supersede RFC-624.

Please let me know if you think the proposal covers your needs, both present and future. I've tried to include watchers representing all possible use cases (different algorithms packages, Middleware, science users), but if you think I've left someone out, please add them.

Concepts

The revised proposal follows the Gen 3 Middleware in making a distinction between bands, which are generic or idealized, and physical filters, which represent specific hardware (for example, HSC-R and HSC-R2 are two physical filters associated with r band). Handling or reporting data by band rather than by physical filter does not carry implications about it being, for example, rigorously calibrated on a photometric system. Any such distinction is outside the scope of the filter system.

The obs package for a given instrument will be the source of truth for that instrument's physical filters, as well as their mapping to bands. We require that all physical filters of observational interest correspond to a band. (This includes narrow-band filters; we show how this could be done in example code, but a naming convention is outside the scope of this RFC.) The system is designed to make it easy to relax this requirement in the future, or to supplement/replace bands with a more complex abstraction later (as Jim Bosch has suggested on occasion).

The system lets algorithmic code identify the band and physical filter associated with a particular exposure, but provides no other functionality. In particular, code that requires wavelength information will now need to get it from a TransmissionCurve object. These are already stored with some Exposures, or can be retrieved from the Butler given a physical filter.

The new system will not support filter aliasing (for example, using W-S-R+ to refer to HSC-R). Each physical filter and each band will be known by exactly one name, as set by the obs package. We expect this change to greatly simplify both filter and client code.

Design

FilterDefinition

Sketch and example code

The existing lsst.obs.base.FilterDefinition and FilterDefinitionCollection classes will be kept, and will continue to define an instrument's physical filters. Like at present, the list of FilterDefinitions will be used to populate both Gen 3 Middleware registries and the initial FilterLabel objects attached to raw exposures.

However, we will get rid of all features specific to lsst.afw.image.Filter and its global registry: aliases, afw_name, wavelength ranges, and the defineFilter(s) methods. Each definition's content will be a mapping from a single physical filter name to an (optional) single band name.

We propose adding a machine-readable documentation field, similar to lsst.pex.config.Field.doc, just because it may be useful in the future if we start populating it now.

FilterLabel

Sketch

An lsst::afw::image::FilterLabel will be a small C++ class that holds the names of the exposure's band and physical filter. It will be populated by the obs package, but will not verify itself against any sort of database or registry thereafter. A FilterLabel may be physical-only (to support "engineering-only" filters) or band-only (to support coadds).

A FilterLabel will be persisted using lsst::afw::table::io::Persistable, as usual for objects attached to an Exposure. The FILTER header keyword will no longer have any special significance, and will be propagated from the original raw data following the usual rules.

While FilterLabel and FilterDefinition store the same information in this proposal, keeping them distinct classes gives us more freedom to let them diverge as our concept of filters evolves (e.g., lots of Middleware-specific stuff added to FilterDefinition that we don't want to impose at the afw level).

I propose that the C++ class be constructable only using factory methods, to maintain flexibility with future changes to the filter system (in particular, new 1-string or 2-string combinations). We could make Python construction use a keyword-only constructor instead of or in addition to this interface.

InstrumentLabel

While not a "filter" system as such, we propose taking the opportunity to add a minimal lsst.afw.table.io.Persistable object to each Exposure that stores the instrument "short name" (equal to lsst.obs.base.Instrument.getName()) associated with an exposure. This would make instrument information as accessible to algorithmic code as physical filter information, and prevent code from needing to assume physical filter names are unique across instruments (though they likely will be in practice). Like all ExposureInfo members, it can simply be omitted when not applicable.

Transition

The replacement of Filter with FilterLabel will be a breaking change to the Exposure persistence format as well as its class methods. I propose the following sequence to implement this RFC:

1. Phase out the use of all filter aliases in LSST code, replacing them with standardized filter names. (DM-27167)
2. Phase out all use of FilterProperty, replacing with e.g. TransmissionCurve. (DM-27168)
3. Implement FilterLabel. (DM-21333)
4. Create a new persistence format (version 2) of ExposureInfo that stores a FilterLabel instead of a Filter. Add new methods to Exposure/ExposureInfo that return/take FilterLabel. Reimplement the existing Filter-based methods to convert to/from FilterLabel internally (this may rearrange the names in the exposed Filter object, but should have no other effects). (DM-27169)
   • So long as Filter exists, we can reproduce old behavior. After Filter is phased out, reading version 1 and older Exposure files will convert the single filter stored there into a band-only FilterLabel if it matches a (short) list of known band names, and into a physical-only FilterLabel otherwise. afw will make no attempt to standardize an assumed physical filter name (e.g., "r2" → "HSC-R2"). (DM-27169)
   • The Middleware will standardize physical filters and bands when reading files in Gen 2 or converting them from Gen 2 to Gen 3 (using the data ID as the source of truth), and astro_metadata_translator will enforce standardized names for all new raw Exposures. This will ensure that old aliases no longer appear after the Gen 3 migration. (DM-27178; AFAIK the Gen 3 side is already implemented)
5. Deprecate Filter and all APIs that use it. (DM-27170)
6. Switch all code that uses Exposure over to FilterLabel. (DM-27170)
7. Switch all other code that uses Filter to FilterLabel. (DM-27170)
8. Remove wavelength info and defineFilters from FilterDefinition. At this point Filter objects are no longer guaranteed to have valid definitions. (DM-27177)
9. Remove the Filter class and supporting infrastructure. (DM-27177)
10. Remove aliases and afw_name from FilterDefinition after Gen 2 is fully retired.