While I have no problem with Tim adding any headers that he feels like to the calibration products, I do object to the idea that these files can now be copied out of the LSST system (in particular the calibration registry) with impunity. That would be an export, and at that point we would add any required metadata (e.g. the validity range, which is not and cannot be tracked by Tim's proposed new keywords).

To put it another way, if we add these keys we should add an explicit rule that no pipeline code may rely on any field in these headers.

First, I should apologise because I've not been following this discussion closely, and what follows may be a dumb question.

However: I'm confused about the distinction between implementation and specification that's implied by this RFC.

It seems totally reasonable to me to insist that calibration products be accompanied by appropriate metadata, and that, when those products are exported from the DM system, that metadata should be packaged with them.

However, it's not obvious why we'd want to legislate that in terms of copying specific headers from one file to another. Why don't we start by writing down what headers are required in the output? Once we've defined that data model, it can then be independent of any particular implementation: whether that metadata is copied from an input file or being pulled from a database or whatever doesn't seem like a very interesting question. And in years to come, when it's all HDF5 or whatever, you won't be scratching your head over obscure rules that are specific to FITS...

I was using examples of FITS headers but since we are writing FITS headers .

The proposal is that output files should always include all the headers from the input files that are the same in all inputs. This is data model independent: If all the files have identical INSTRUME headers then we get an INSTRUME in the output file. Output files will have headers in them even if they are written in HDF5 format.

Additionally I would like some information in the output files to indicate the range of conditions and times covering the processed product. My proposal was that the simplest way to do this is to obtain them directly from the input files (oldest and newest) and use a config item to indicate which keys are relevant for the particular instrument. This is a general approach that does not require that each camera share the same headers. In the past I've done this for many time dependent headers to indicate azimuth/elevation start/end, airmass start/end, weather conditions at start and end. I wasn't asking for the list of headers on this RFC, I was more seeing if people objected to the idea in general.

pipe_drivers currently attempts to use VisitInfo to generate LSST-specific output headers but it only does that for dark time and exposure time. If pipe_drivers calibration construction was rewritten to calculate these range values from ObservationInfo and afw.image.setVisitInfoMetadata was replaced with something that understood a collection of ObservationInfo, then the DATE-OBS/DATE-END information could be derived directly without requiring that we get it from the raw data (so no need to specify instrument-specific time-dependent headers). We currently have more time-dependent headers defined in LSE-400 than are present in ObservationInfo (and many more than are in VisitInfo) but that approach might work.

John Swinbank would you be happier with a modified proposal whereby all the constant value headers are propagated and time-dependent values are handled via ObservationInfo collections in a general way?

Hi Tim Jenness — just to be clear, I'm not actually unhappy with this proposal, just a little unsure of my ground.

It seems surprising to me that we'd define the metadata for a product by naïvely propagating the metadata of its inputs, rather than by writing down what the required metadata for that type of product is. But... that's “surprising”, rather than “wrong” or “upsetting” — I'm totally prepared to believe this is the right way to do things, either as a matter of safety (ie, being sure you don't miss anything) or of expediency (it's quicker & easier to do this than to have some master document where we document the metadata needed for all of our products).

Perhaps some of the thinking about what metadata is necessary where has already been done in LSE-400? Unfortunately, I can't find that (it's not at ls.st/lse-400*, https://lse-400.lsst.io/, https://github.com/lsst/lse-400, or https://www.lsst.io/#lse), so it doesn't help me much.

I also don't object to your proposal to selecting DATE-OBS and DATE-END headers. It does seem like this is something that should be defined in a document somewhere, though, rather than just as a matter of implementation.

Speaking of implementation, I entirely defer to you (and Robert, and whoever else feels strongly about it) about the mechanism by which this metadata is propagated; I'm sure you understand that much better than I do.

Adopting this as:

1. Any headers that are present in all inputs with identical values will appear in the output file.
2. I will expand the VisitInfo to Header behavior such that standardized information can come from the earliest and latest input files.

pipe_drivers has been updated so that header propagation now occurs for calibration products. I will mark this RFC as implemented but note that this RFC also covers generation of tracts/patches and the changes to pipe_drivers must end up in the new calibration products pipeline.