Details
-
Type:
RFC
-
Status: Adopted
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
The "internal file format" for images has purposely been left undefined to allow us to choose the best available format to support our needs. It has always been a requirement for us to be able to export images as FITS files, although it is not absolutely required that all available information be placed in those FITS files (so that they could be "round-tripped").
It is becoming urgent for us to define what the image interface is between the DAX and SUIT teams. Since SUIT uses Java, the Butler and afw are not available to help intermediate. SUIT needs a serialized file format that is efficient to access from Java.
Mario and I propose that we baseline the choice of FITS as the image interface. This is what is currently supported on the SUIT side and already required to be output by the DAX side, so it seems like the lowest-cost choice.
The Science Pipelines do not need this level of baselining because they have the Butler to isolate them from the serialized file format. The Butler should still maintain the ability to support multiple serializations.
If future developments in file formats provide additional functionality that SUIT could use, this decision could be revisited.
Attachments
Issue Links
- is triggering
-
- To Do
Activity
could camera archiving and prompt processing please tailgate on this RFC? Archived data acquired from the spectrograph, comcam main cam data. also files presented to the prompt processing system?
The rationale is similar.
Donald Petravick
added a comment - could camera archiving and prompt processing please tailgate on this RFC? Archived data acquired from the spectrograph, comcam main cam data. also files presented to the prompt processing system?
The rationale is similar. It would be helpful to both users and SUIT if images returned from DAX queries, in FITS (or other future) file format, included metadata identifying - if applicable - their origin from the known LSST image data products (e.g., raw, calexp, single-band coadd, multi-band coadd) and whether their pixel content is taken 1:1 from those data products (e.g., an integral x,y region taken from a calexp) or resampled (by rotation, offset, or scaling). It would also be useful if the image contained metadata (a "header", to get all primitive about it) providing a recipe (URI?) that would permit re-requesting it in the future.
Thought should be given to whether there is a way to ensure that both these forms of metadata are not misleadingly preserved across reading the image in, modifying its pixels, and writing it back out. Tim Jenness or Frossie Economou may be able to remind us of an earlier system that achieved this.
I'm okay with this proposal on the assumption that we aren't talking about the metadata issues presented by Gregory Dubois-Felsmann.
There doesn't seem to be any disagreement about the core of this proposal. I will implement it as a "physical representation" addendum to what is expected to be an upcoming "DM Interface Data Model" document which would point to a database like the current schema browser back-end and like Wil's Gaia data model database.
As to Don's piggybacking, my thoughts are:
- While FITS may be a pragmatic choice for file format, it's not clear to me that any of the above arguments apply: this is code we are writing without a long history, and it uses Python and so potentially has access to all of our tools.
- I remain a bit worried about the effect on latency of multiple data copies (even if not to disk) and transformations between the Prompt Processing harness and the science payload. Preserving the option to pass the DAQ's delivered data in as "raw" a form as possible might be desirable.
So I would rather not decide those issues at this time, leaving them for another RFC.
I believe that an adequate implementation of this will be by adding this to LDM-463.