Description
DMS-REQ-0078 states
The DMS catalog archive shall provide catalog data and associated metadata on request in community standard formats: * Comma-separated ASCII text * eXtensible Markup Language (XML) format, including VOTable (http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOTable), and * FITS tables.
There are a couple of issues here:
- The VOTable reference, to the IVOA wiki, is not concrete enough to be verifiable, and should be changed to a reference to a specific version of the standard - I suggest VOTable 1.4. The corresponding LSP requirement can address, if we wish, finer-grained details such as which data representations within VOTable are required (e.g., TABLEDATA and BINARY2).
- The requirement for FITS table access has not been addressed particularly carefully in our design, and it is possible that it has been superseded by RFC-662 and the contemporary focus on Parquet.
Concretely, I am proposing to update the normative text to require VOTable 1.4.
Beyond that, we need to discuss whether to retain the FITS table catalog access requirement, and if so, we need to confirm that as part of the design baseline and determine whether that requirement is allocated to the TAP service or to the bulk download service or both.
We also need to determine whether Parquet access will be left just as a design feature or whether it will be added to LSE-61. (DM-24548 already calls for it to be added to the DPDD baseline.)
Specific proposal:
LSE-61 (DMSR)
EXISTING
4.1.15 Catalog Export Formats
ID: DMS-REQ-0078 (Priority: 1a)
Specification: The DMS catalog archive shall provide catalog data and associated metadata on request in community standard formats: * Comma-separated ASCII text * eXtensible Markup Language (XML) format, including VOTable (http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOTable), and * FITS tables.
(no Discussion)
PROPOSED
4.1.15 Catalog Export Formats
ID: DMS-REQ-0078 (Priority: 1a)
Specification: The DMS catalog archive shall provide catalog data and associated metadata on request in community standard formats:
- Comma-separated ASCII text (CSV);
- IVOA VOTable format, version 1.4 or later, supporting at least the XML-based TABLEDATA payload format;
- FITS tables.
Discussion: The VOTable 1.4 standard is available at http://www.ivoa.net/Documents/VOTable/20191021 . It is likely that additional payload formats will be supported, including BINARY2. "Associated metadata" for catalogs should include catalog schema and temporal and spatial coordinate-system definitions. Full metadata will likely only be available in the VOTable format.
LDM-554 (LSP Requirements)
EXISTING
4.4.1.1 VOTable Output for TAP
ID: DMS-API-REQ-0010
Specification: The API Aspect TAP endpoint shall support IVOA VOTable 1.3 as an available output format
PROPOSED
4.4.1.1 VOTable Output from API Aspect Services
ID: DMS-API-REQ-0010
Specification: API Aspect services based on IVOA standards (e.g., TAP, SIAv2) which return tabular data shall support IVOA VOTable, version 1.4 or later, as their default output format, unless otherwise specified in the applicable protocol standard.
Discussion: VOTable 1.4 is specified in order to support the return of TIMESYS and COOSYS elements defining the coordinate systems used. The "unless otherwise specified" provides an exception for Simple Cone Search, which is written in a way that formally doesn't permit VOTable versions beyond 1.1 (fixing is the subject of a proposed change to SCS).
This looks good to me. Thank you.