Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: afw
-
Story Points:1
-
Epic Link:
-
Sprint:DRP F16-3
-
Team:Data Release Production
Description
In looking at the Source table outputs (SRC*.fits files) in the validation_data_hsc repository, we noticed that the headers are not consistently applied in the FITS files generated by afw.table output. In particular, the TUNIT keywords are missing for the coord_ra and coord_dec columns, making it difficult for a client to determine that the columns are persisted in units of radians, especially since the FITS convention is for the default units for angles to be degrees.
KEYWORD: TUNITn
REFERENCE: FITS Standard
STATUS: reserved
HDU: table
VALUE: string
COMMENT: column units
DEFINITION: The value field shall contain a character string describing
the physical units in which the quantity in field n, after any
application of TSCALn and TZEROn, is expressed. The units of all FITS
header keyword values, with the exception of measurements of angles,
should conform with the recommendations in the IAU Style Manual. For
angular measurements given as floating point values and specified with
reserved keywords, degrees are the recommended units (with the units,
if specified, given as 'deg').
http://heasarc.gsfc.nasa.gov/docs/fcg/standard_dict.html
Accepting that we are not conforming to that convention, the proper value for TUNITn is "rad" for data in units of radians.
While much access to afw.table-produced files will be through the afw.table interfaces, where this issue is irrelevant as long as round-tripping is successful, for exploratory and debugging purposes it is very useful to be able to access the file contents with standard tools.
In addition, if the FITS tables produced from afw.table are to be used as the input to the database ingest, providing units for all columns will assist the database system in providing and validating correct column metadata to its clients, whether through VO standards or other means.
Attachments
Issue Links
Activity
Master. We asked for up-to-date sample outputs of the pipelines in order to exercise Firefly, and we were just pointed at the repo.
I don't think it's changed, but please verify with some data that has been processed with a recent version of the stack.
Paul Price
added a comment - I don't think it's changed, but please verify with some data that has been processed with a recent version of the stack. Found the bug; fixing it now.
Just as a note of a side-conversation with Jim Bosch:
I also noticed that there are thousands of duplicate AR_NAME headers in some tables, and mentioned this to Jim. We are adding that as an additional bug report on this ticket.
Ready for review; John Parejko, could you take a look?
This is three separate fixes all dealing with afw.table persistence, split into three commits (with another one that's purely cleanup). In addition to the two changes (angle unit writing and AR_NAME header key duplication) described above, I've also modified the angle-column reading code to better handle non-LSST and/or malformed inputs. All changes are on tickets/DM-7221 of afw.
Temporarily taking this out of review, because Jenkins doesn't like it (looks like I broke another test that I hadn't been re-running locally).
Fixed, and once again ready for review. I scaled by my ambitions on reading columns with angle units to only read those tagged specifically as Angles by our code back into Angle fields; converting fields without those tags was backwards incompatible, and hence a bigger change than I should attempt on this issue.
Just a couple of comments in the PR, but otherwise it looks good.
Which branch did you look at in validation_data_hsc? The processed data on the master branch was created a while ago, before we got serious about units. If that's what you're basing this report on, please look at data that has been processed recently, e.g., by running ci_hsc or looking in the tickets/
DM-6048branch of validation_data_hsc (warning: 300 GB).