Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_drivers
-
Labels:None
-
Story Points:4
-
Epic Link:
-
Sprint:Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11, Arch 2019-03-18
-
Team:Architecture
Description
I was testing the new metadata translator on FITS file I had lying around in testdata repositories and I see that our calibration products do not seem to have a reasonable set of headers. The only indication this is a HSC calibration file rather than an LSST one is the path name to the root directory.
SIMPLE = T / file does conform to FITS standard
|
BITPIX = -32 / number of bits per data pixel
|
NAXIS = 2 / number of data axes
|
NAXIS1 = 2048 / length of data axis 1
|
NAXIS2 = 4176 / length of data axis 2
|
EXTEND = T / FITS dataset may contain extensions
|
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
|
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
|
OBSTYPE = 'bias '
|
HIERARCH CALIB_CREATION_DATE = '2016-03-28'
|
HIERARCH CALIB_CREATION_TIME = '16:16:33 EDT'
|
HIERARCH CALIB_CREATION_ROOT = '/tigress/HSC/HSC/rerun/dm-5124/calib'
|
HIERARCH CALIB_INPUT_0 = '(904542,)'
|
HIERARCH CALIB_INPUT_1 = '(904544,)'
|
HIERARCH CALIB_INPUT_2 = '(904546,)'
|
HIERARCH CALIB_INPUT_3 = '(904548,)'
|
HIERARCH CALIB_INPUT_4 = '(904550,)'
|
HIERARCH CALIB_INPUT_5 = '(904552,)'
|
HIERARCH CALIB_INPUT_6 = '(904554,)'
|
HIERARCH CALIB_INPUT_7 = '(904556,)'
|
HIERARCH CALIB_INPUT_8 = '(904558,)'
|
HIERARCH CALIB_INPUT_9 = '(904560,)'
|
HIERARCH CALIB_INPUT_10 = '(904562,)'
|
HIERARCH CALIB_INPUT_11 = '(904564,)'
|
HIERARCH CALIB_INPUT_12 = '(904566,)'
|
CALIB_ID= 'filter=NONE calibDate=2013-11-03 ccd=50'
|
HIERARCH MD5_IMAGE = '6185bb72f20de7e81c45e6e6591eb6ad'
|
CRVAL1A = 0 / Column pixel of Reference Pixel
|
CRVAL2A = 0 / Row pixel of Reference Pixel
|
CRPIX1A = 1 / Column Pixel Coordinate of Reference
|
CRPIX2A = 1 / Row Pixel Coordinate of Reference
|
CTYPE1A = 'LINEAR ' / Type of projection
|
CTYPE2A = 'LINEAR ' / Type of projection
|
CUNIT1A = 'PIXEL ' / Column unit
|
CUNIT2A = 'PIXEL ' / Row unit
|
At minimum there should be INSTRUME/TELESCOP headers but ideally we should be able to construct a full header based on all the input headers (propagating headers that are identical and indicating which headers define bounds). The latter is what I implemented at the JCMT Science Archive for all data products. It might be that another approach is to at least be able to generate a standard set of headers from the information the Butler (Gen 3) knows about each dataset.
Attachments
Issue Links
Activity
This was easier than I expected (to at least prototype). Changes needed to pipe_drivers and astro_metadata_translator but now the auxTel ci_lsst calibration output looks like:
SIMPLE = T / file does conform to FITS standard
|
BITPIX = 8 / number of bits per data pixel
|
NAXIS = 0 / number of data axes
|
EXTEND = T / FITS dataset may contain extensions
|
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
|
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
|
HEADVER = 2 / Version number of header
|
INSTRUME= 'LSST_ATISS' / Instrument
|
TELESCOP= 'LSST_AuxTel' / Telescope
|
SEQFILE = 'ats_20180511.seq' / Sequencer file name
|
CCD_MANU= 'ITL ' / CCD Manufacturer
|
CCD_TYPE= '3800C ' / CCD Model Number
|
CCD_SERN= '20304 ' / Manufacturers' CCD Serial Number
|
LSST_NUM= 'ITL-3800C-098' / LSST Assigned CCD Number
|
DETSIZE = '[1:4072,1:4000]' / IRAF detector size
|
EXPTIME = 0. / Exposure Time in Seconds
|
TELCODE = 'AT ' / The "code" for AuxTel
|
CONTRLLR= 'C ' / The controller (e.g. O for OCS, C for CCS)
|
DAYOBS = '20180920' / The observation day as defined in the image nam
|
MJD-OBS = 58382.2355372338 / Modified Julian Date of image acquisition
|
DATE-OBS= '2018-09-21T05:39:10.417' / Date of the observation (image acquisition
|
DARKTIME= 0.
|
ROTTYPE = 'UNKNOWN ' / Type of rotation angle
|
OBSTYPE = 'bias '
|
HIERARCH CALIB_CREATION_DATE = '2019-02-15'
|
HIERARCH CALIB_CREATION_TIME = '15:13:07 MST'
|
DATE-AVG= '2018-09-21T00:00:00.00'
|
HIERARCH CALIB_INPUT_0 = '(2018092000028,)'
|
HIERARCH CALIB_INPUT_1 = '(2018092000029,)'
|
HIERARCH CALIB_INPUT_2 = '(2018092000030,)'
|
HIERARCH CALIB_INPUT_3 = '(2018092000031,)'
|
HIERARCH CALIB_INPUT_4 = '(2018092000032,)'
|
HIERARCH CALIB_INPUT_5 = '(2018092000033,)'
|
HIERARCH CALIB_INPUT_6 = '(2018092000034,)'
|
HIERARCH CALIB_INPUT_7 = '(2018092000035,)'
|
HIERARCH CALIB_INPUT_8 = '(2018092000036,)'
|
CALIB_ID= 'detector=0 detectorName=S00 filter=NONE calibDate=2018-09-21'
|
HIERARCH MD5_IMAGE = 'eda3f77dfa7ea894a0f796e24615a42a'
|
MD5_MASK= 'fb394efc9596d6865b4dc311839da4f9'
|
HIERARCH MD5_VARIANCE = '8df0c2327f53ee09cb391743817c3f94'
|
HIERARCH EXPINFO_V = 0
|
AR_HDU = 5 / HDU (1-indexed) containing the archive used to
|
FILTER = '_unknown_'
|
FLUXMAG0= 0.
|
HIERARCH FLUXMAG0ERR = 0.
|
END
|
Putting extra information into the headers is fine, although we don't need it for LSST. The butler knows where to find the data, and the full list of input data used to generate the calibs is already in the metadata.
I am more concerned about the DATE-OBS and DATE-END fields. These values don't tell us about the validity range of the calibs (that's set upon ingest into the calib registry), and these keys might imply the contrary.
Following on from RFC-576, I've reworked the pull request to propagate headers without trying to forward first/last values of some special headers. For that I will have a go at reworking the VisitInfo logic in pipe_drivers so that I can get the date-start/end values from VisitInfo. I might fix the "average date" value as well since it seems wrong at the moment.
Paul Price would you be able to look at this? It's a small change to pipe_drivers constructCalibs.py to propagate input headers to outputs and to add some standard date headers. The metadata mangling itself is handed off to astro_metadata_translator.
pipe_drivers changes look good. Please be sure to test them, as they do not currently get exercised by unit tests, the demo or ci_hsc. I think ci_ctio0m9 might do.
Paul Price
added a comment - pipe_drivers changes look good. Please be sure to test them, as they do not currently get exercised by unit tests, the demo or ci_hsc. I think ci_ctio0m9 might do. They work with ci_lsst (Which is where I test them). I will have a look at ci_ctio0m9.
Simon Krughoff thank you for agreeing to review the changes to astro_metadata_translator.
As I understand it Paul Price is ok with the pipe_drivers changes. I've reviewed the other changes and have a few comments on the PR.
Thanks for the reviews. I tested with ci_ctio0m9 and tweaked the patch so that it functions properly if header translation fails. Jenkins passed and I tested again with ci_lsst. Merged.
We should also consider whether DATE-OBS and DATE-END in the outputs should reflect the earliest DATE-OBS and latest DATE-END of the inputs.