Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-255

Define FITS to be the image interface between DAX and SUIT

    XMLWordPrintable

Details

    • RFC
    • Status: Adopted
    • Resolution: Done
    • DM
    • None

    Description

      The "internal file format" for images has purposely been left undefined to allow us to choose the best available format to support our needs. It has always been a requirement for us to be able to export images as FITS files, although it is not absolutely required that all available information be placed in those FITS files (so that they could be "round-tripped").

      It is becoming urgent for us to define what the image interface is between the DAX and SUIT teams. Since SUIT uses Java, the Butler and afw are not available to help intermediate. SUIT needs a serialized file format that is efficient to access from Java.

      Mario and I propose that we baseline the choice of FITS as the image interface. This is what is currently supported on the SUIT side and already required to be output by the DAX side, so it seems like the lowest-cost choice.

      The Science Pipelines do not need this level of baselining because they have the Butler to isolate them from the serialized file format. The Butler should still maintain the ability to support multiple serializations.

      If future developments in file formats provide additional functionality that SUIT could use, this decision could be revisited.

      Attachments

        Issue Links

          Activity

            ktl Kian-Tat Lim added a comment -

            I believe that an adequate implementation of this will be by adding this to LDM-463.

            ktl Kian-Tat Lim added a comment - I believe that an adequate implementation of this will be by adding this to LDM-463.

            could camera archiving and prompt processing please tailgate on this RFC? Archived data acquired from the spectrograph, comcam main cam data. also files presented to the prompt processing system?

            The rationale is similar.

            petravick Donald Petravick added a comment - could camera archiving and prompt processing please tailgate on this RFC? Archived data acquired from the spectrograph, comcam main cam data. also files presented to the prompt processing system? The rationale is similar.

            It would be helpful to both users and SUIT if images returned from DAX queries, in FITS (or other future) file format, included metadata identifying - if applicable - their origin from the known LSST image data products (e.g., raw, calexp, single-band coadd, multi-band coadd) and whether their pixel content is taken 1:1 from those data products (e.g., an integral x,y region taken from a calexp) or resampled (by rotation, offset, or scaling). It would also be useful if the image contained metadata (a "header", to get all primitive about it) providing a recipe (URI?) that would permit re-requesting it in the future.

            Thought should be given to whether there is a way to ensure that both these forms of metadata are not misleadingly preserved across reading the image in, modifying its pixels, and writing it back out. tjenness or frossie may be able to remind us of an earlier system that achieved this.

            gpdf Gregory Dubois-Felsmann added a comment - It would be helpful to both users and SUIT if images returned from DAX queries, in FITS (or other future) file format, included metadata identifying - if applicable - their origin from the known LSST image data products (e.g., raw, calexp, single-band coadd, multi-band coadd) and whether their pixel content is taken 1:1 from those data products (e.g., an integral x,y region taken from a calexp) or resampled (by rotation, offset, or scaling). It would also be useful if the image contained metadata (a "header", to get all primitive about it) providing a recipe (URI?) that would permit re-requesting it in the future. Thought should be given to whether there is a way to ensure that both these forms of metadata are not misleadingly preserved across reading the image in, modifying its pixels, and writing it back out. tjenness or frossie may be able to remind us of an earlier system that achieved this.
            tjenness Tim Jenness added a comment -

            I'm okay with this proposal on the assumption that we aren't talking about the metadata issues presented by gpdf.

            tjenness Tim Jenness added a comment - I'm okay with this proposal on the assumption that we aren't talking about the metadata issues presented by gpdf .

            There doesn't seem to be any disagreement about the core of this proposal. I will implement it as a "physical representation" addendum to what is expected to be an upcoming "DM Interface Data Model" document which would point to a database like the current schema browser back-end and like Wil's Gaia data model database.

            As to Don's piggybacking, my thoughts are:

            • While FITS may be a pragmatic choice for file format, it's not clear to me that any of the above arguments apply: this is code we are writing without a long history, and it uses Python and so potentially has access to all of our tools.
            • I remain a bit worried about the effect on latency of multiple data copies (even if not to disk) and transformations between the Prompt Processing harness and the science payload. Preserving the option to pass the DAQ's delivered data in as "raw" a form as possible might be desirable.

            So I would rather not decide those issues at this time, leaving them for another RFC.

            ktl Kian-Tat Lim added a comment - There doesn't seem to be any disagreement about the core of this proposal. I will implement it as a "physical representation" addendum to what is expected to be an upcoming "DM Interface Data Model" document which would point to a database like the current schema browser back-end and like Wil's Gaia data model database. As to Don's piggybacking, my thoughts are: While FITS may be a pragmatic choice for file format, it's not clear to me that any of the above arguments apply: this is code we are writing without a long history, and it uses Python and so potentially has access to all of our tools. I remain a bit worried about the effect on latency of multiple data copies (even if not to disk) and transformations between the Prompt Processing harness and the science payload. Preserving the option to pass the DAQ's delivered data in as "raw" a form as possible might be desirable. So I would rather not decide those issues at this time, leaving them for another RFC.

            People

              ktl Kian-Tat Lim
              ktl Kian-Tat Lim
              Colin Slater, Donald Petravick, Gregory Dubois-Felsmann, John Parejko, Kian-Tat Lim, Tim Jenness, Xiuqin Wu [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.