Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15783

Recognize and exploit ObsCore-structured data in Firefly



    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Firefly, SUIT
    • Labels:
    • Team:
      Science User Interface


      As part of the developing strategy for designing clearer interfaces into the Science Platform:

      Firefly should recognize tabular data in ObsCore format and expose appropriate behavior when it is received in any form carrying sufficient metadata for it to be recognized.

      The result should not be strongly dependent on whether the ObsCore data are received via a "manual" table upload via the Firefly API, the opening of a file (e.g., from a workspace) in this format, or via the normal Firefly search processor API.

      This ticket covers only recognizing the table and providing the behavior, not the provision of any search screens or other interfaces that assist the user in finding/retrieving ObsCore-formatted data. However, it is worth noting here that implementing this ticket is a key predecessor to providing good support in Firefly for SIAv2 and ObsTAP searches, in particular.

      It should also be noted that the work in this ticket is a prerequisite to providing decent initial support for using Firefly to browse LSST image data, and associated data products, both in Notebook-based exploratory scenarios where the data may be "pushed" at Firefly, and in LSP Portal scenarios where a Portal search screen leads the user to this capability.

      This capability is also a predecessor and sort of a dry run for full support of CAOM-oriented observation data in Firefly. It should also be regarded as a generalization and improvement upon Firefly's existing "four corners" support for image metadata tables.

      This tickets defines ObsCore as a sort of minimal interface for any Firefly applications that involve the display of tables of observation metadata and as such can then be taken advantage of in new applications simply by meeting that interface. This may be relevant for, e.g., SOFIA and other new IRSA projects.

      1) What is the "appropriate behavior"?

      Priority 1: the most essential parts of this are:

      1. Recognizing the "s_ra" and "s_dec" columns (UCDs "pos.eq.ra" and "pos.eq.dec") and displaying all of the corresponding points, if not nulled, in a context image (especially a HiPS image). A "blank" context image should be supported eventually, but this is not required at first.
      2. Recognizing the "s_region" column in an ObsCore data table and successfully displaying it, if not nulled, as a geometrical region in a context image (especially a HiPS image). It should be API-selectable whether all regions are always displayed, with the highlighted row in the table having its region highlighted in the context image, or whether only the highlighted row has its region displayed at all.
      3. Recognizing the "access_url" column, displaying it as a link, and enabling the link to be used for a download of the associated data file.
      4. Recognizing "dataproduct_type=image" rows and loading and displaying the associated images if the "access_format" value suggests that Firefly should understand the image data format (FITS, HiPS, or a browser-displayable image such as JPEG, PNG, SVG).
      5. There must be API support for supplying the context image to be used. (It may be useful to design the system so that a context image could be inferred from data in the table itself, e.g., the "facility_name", "instrument_name", and "obs_collection". That is not a Priority 1 requirement, though.)
      6. This ticket does not mandate providing any search-form support for SIAv2 or ObsTAP. However, if a valid URL for a method=GET synchronous search on such a service, returning valid ObsCore data, is provided to Firefly in the existing "Upload table from URL" screen it must be recognized as ObsCore and displayed appropriately. This is a key feature for testing as well as for demonstrations of the capability.

      Priority 2:

      1. Recognizing rows that refer to tabular data types likely to arise in LSST data. Highest priority is "dataproduct_type=measurements" data in the form of FITS tables. It should be possible to invoke the "FITS can opener" on such files and display the tables. I will try to address finer details of this either in comments or in associated documentation.

      Priority 3:

      1. (To be filed as a separate ticket) Providing a "property sheet" for ObsCore table rows, providing a nicely formatted per-observation page. This could be the "pilot case" for building the Firefly infrastructure for formatted property sheets.
      2. Recognizing the "s_fov" value, if not nulled, and allowing the display of the associated circle in the context image.
      3. Providing support for filtering on "t_min" and "t_max" with human-readable dates, not just MJD values.
      4. Recognizing when the table also contains DataLink metadata and offer appropriate "row actions"; to be written up separately.

      Ideas for later:

      1. Supporting setting up cone searches based on "s_ra, s_dec, and s_fov".
      2. Supporting setting up polygon, etc., searches based on supported types of "s_region".

      2) What is "sufficient metadata"?

      1. The reference use case is for the application of this functionality to VOTable data containing FIELDs with ObsCore names, UCDs, and utypes.
      2. It would be useful to be able to recognize data in other formats with less explicit data models but with the presence of the mandatory column names that would strongly suggest an ObsCore data model (obs_id, dataproduct_type, calib_level, s_ra, s_dec, access_url). That way a round trip through a less rich table format would not destroy the main ability to handle the data.

      The idea here is that, upon ingest, a table would be tagged with a "likely to be ObsCore" flag that would enable all the above behavior.

      The presence of additional columns beyond the ObsCore minimum should not interfere with offering the above behavior.

      3) Prerequisites

      1. (pretty much mandatory) Ingest, preserve, and expose via internal APIs the VOTable metadata: at a very minimum the <FIELD> metadata.
      2. (highly desirable) Recognize the presence of category columns in displayed tables and provide a "checkbox" interface for filtering them, allowing, e.g., a two-click reduction of an ObsCore table to only the "dataproduct_type = image" rows. This lets us avoid providing any custom UI elements (like "show only images") for exploring ObsCore data (though such things might be desirable in the more distant future for a more elegant interface).

      4) Backward compatibility issues

      Since this is expected to become a "core" capability of Firefly, it will/should automatically appear when ObsCore data are loaded under any circumstances. In the long run, this is a good thing (IMHO) but in the short run it might produce surprising results in certain circumstances with existing searches. (Perhaps not all that likely given what I know about IRSA's applications, but this is an assertion that should be reviewed by actual IRSA staff.)

      Therefore, it may be reasonable for there to be an application-level switch allowing the suppression of the automatic special behavior for ObsCore. This should be checked with IRSA and NED.


          Issue Links


            gpdf Gregory Dubois-Felsmann added a comment -

            Xiuqin Wu [X] should we move this to a FIREFLY ticket?

            gpdf Gregory Dubois-Felsmann added a comment - Xiuqin Wu [X] should we move this to a FIREFLY ticket?
            xiuqin Xiuqin Wu [X] (Inactive) added a comment -

            Only one child ticket was not implemented. All others have been implemented and the Firefly tickets are listed in DM-18510. I think only DM-18692 needs to be duplicated in Firefly. 

            xiuqin Xiuqin Wu [X] (Inactive) added a comment - Only one child ticket was not implemented. All others have been implemented and the Firefly tickets are listed in DM-18510 . I think only DM-18692 needs to be duplicated in Firefly. 


              xiuqin Xiuqin Wu [X] (Inactive)
              gpdf Gregory Dubois-Felsmann
              Christine Banek, Frossie Economou, Gregory Dubois-Felsmann, Loi Ly, Tim Jenness, Trey Roby, Vandana Desai, Xiuqin Wu [X] (Inactive)
              0 Vote for this issue
              8 Start watching this issue




                  No builds found.