Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16292

Calibration products need more metadata

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_drivers
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11, Arch 2019-03-18
    • Team:
      Architecture

      Description

      I was testing the new metadata translator on FITS file I had lying around in testdata repositories and I see that our calibration products do not seem to have a reasonable set of headers. The only indication this is a HSC calibration file rather than an LSST one is the path name to the root directory.

      SIMPLE  =                    T / file does conform to FITS standard             
      BITPIX  =                  -32 / number of bits per data pixel                  
      NAXIS   =                    2 / number of data axes                            
      NAXIS1  =                 2048 / length of data axis 1                          
      NAXIS2  =                 4176 / length of data axis 2                          
      EXTEND  =                    T / FITS dataset may contain extensions            
      COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
      COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
      OBSTYPE = 'bias    '                                                            
      HIERARCH CALIB_CREATION_DATE = '2016-03-28'                                     
      HIERARCH CALIB_CREATION_TIME = '16:16:33 EDT'                                   
      HIERARCH CALIB_CREATION_ROOT = '/tigress/HSC/HSC/rerun/dm-5124/calib'           
      HIERARCH CALIB_INPUT_0 = '(904542,)'                                            
      HIERARCH CALIB_INPUT_1 = '(904544,)'                                            
      HIERARCH CALIB_INPUT_2 = '(904546,)'                                            
      HIERARCH CALIB_INPUT_3 = '(904548,)'                                            
      HIERARCH CALIB_INPUT_4 = '(904550,)'                                            
      HIERARCH CALIB_INPUT_5 = '(904552,)'                                            
      HIERARCH CALIB_INPUT_6 = '(904554,)'                                            
      HIERARCH CALIB_INPUT_7 = '(904556,)'                                            
      HIERARCH CALIB_INPUT_8 = '(904558,)'                                            
      HIERARCH CALIB_INPUT_9 = '(904560,)'                                            
      HIERARCH CALIB_INPUT_10 = '(904562,)'                                           
      HIERARCH CALIB_INPUT_11 = '(904564,)'                                           
      HIERARCH CALIB_INPUT_12 = '(904566,)'                                           
      CALIB_ID= 'filter=NONE calibDate=2013-11-03 ccd=50'                             
      HIERARCH MD5_IMAGE = '6185bb72f20de7e81c45e6e6591eb6ad'                         
      CRVAL1A =                    0 / Column pixel of Reference Pixel                
      CRVAL2A =                    0 / Row pixel of Reference Pixel                   
      CRPIX1A =                    1 / Column Pixel Coordinate of Reference           
      CRPIX2A =                    1 / Row Pixel Coordinate of Reference              
      CTYPE1A = 'LINEAR  '           / Type of projection                             
      CTYPE2A = 'LINEAR  '           / Type of projection                             
      CUNIT1A = 'PIXEL   '           / Column unit                                    
      CUNIT2A = 'PIXEL   '           / Row unit                                      
      

      At minimum there should be INSTRUME/TELESCOP headers but ideally we should be able to construct a full header based on all the input headers (propagating headers that are identical and indicating which headers define bounds). The latter is what I implemented at the JCMT Science Archive for all data products. It might be that another approach is to at least be able to generate a standard set of headers from the information the Butler (Gen 3) knows about each dataset.

        Attachments

          Issue Links

            Activity

            No builds found.
            tjenness Tim Jenness created issue -
            Hide
            tjenness Tim Jenness added a comment -

            We should also consider whether DATE-OBS and DATE-END in the outputs should reflect the earliest DATE-OBS and latest DATE-END of the inputs.

            Show
            tjenness Tim Jenness added a comment - We should also consider whether DATE-OBS and DATE-END in the outputs should reflect the earliest DATE-OBS and latest DATE-END of the inputs.
            swinbank John Swinbank made changes -
            Field Original Value New Value
            Remote Link This issue links to "Page (Confluence)" [ 19479 ]
            tjenness Tim Jenness made changes -
            Component/s pipe_drivers [ 13309 ]
            tjenness Tim Jenness made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Assignee Tim Jenness [ tjenness ]
            tjenness Tim Jenness made changes -
            Sprint Arch 2019-02-11 [ 862 ]
            Story Points 2
            Team Architecture [ 10304 ]
            Hide
            tjenness Tim Jenness added a comment -

            This was easier than I expected (to at least prototype). Changes needed to pipe_drivers and astro_metadata_translator but now the auxTel ci_lsst calibration output looks like:

            SIMPLE  =                    T / file does conform to FITS standard             
            BITPIX  =                    8 / number of bits per data pixel                  
            NAXIS   =                    0 / number of data axes                            
            EXTEND  =                    T / FITS dataset may contain extensions            
            COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
            COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
            HEADVER =                    2 / Version number of header                       
            INSTRUME= 'LSST_ATISS'         / Instrument                                     
            TELESCOP= 'LSST_AuxTel'        / Telescope                                      
            SEQFILE = 'ats_20180511.seq'   / Sequencer file name                            
            CCD_MANU= 'ITL     '           / CCD Manufacturer                               
            CCD_TYPE= '3800C   '           / CCD Model Number                               
            CCD_SERN= '20304   '           / Manufacturers' CCD Serial Number               
            LSST_NUM= 'ITL-3800C-098'      / LSST Assigned CCD Number                       
            DETSIZE = '[1:4072,1:4000]'    / IRAF detector size                             
            EXPTIME =                   0. / Exposure Time in Seconds                       
            TELCODE = 'AT      '           / The "code" for AuxTel                          
            CONTRLLR= 'C       '           / The controller (e.g. O for OCS, C for CCS)     
            DAYOBS  = '20180920'           / The observation day as defined in the image nam
            MJD-OBS =     58382.2355372338 / Modified Julian Date of image acquisition      
            DATE-OBS= '2018-09-21T05:39:10.417' / Date of the observation (image acquisition
            DARKTIME=                   0.                                                  
            ROTTYPE = 'UNKNOWN '           / Type of rotation angle                         
            OBSTYPE = 'bias    '                                                            
            HIERARCH CALIB_CREATION_DATE = '2019-02-15'                                     
            HIERARCH CALIB_CREATION_TIME = '15:13:07 MST'                                   
            DATE-AVG= '2018-09-21T00:00:00.00'                                              
            HIERARCH CALIB_INPUT_0 = '(2018092000028,)'                                     
            HIERARCH CALIB_INPUT_1 = '(2018092000029,)'                                     
            HIERARCH CALIB_INPUT_2 = '(2018092000030,)'                                     
            HIERARCH CALIB_INPUT_3 = '(2018092000031,)'                                     
            HIERARCH CALIB_INPUT_4 = '(2018092000032,)'                                     
            HIERARCH CALIB_INPUT_5 = '(2018092000033,)'                                     
            HIERARCH CALIB_INPUT_6 = '(2018092000034,)'                                     
            HIERARCH CALIB_INPUT_7 = '(2018092000035,)'                                     
            HIERARCH CALIB_INPUT_8 = '(2018092000036,)'                                     
            CALIB_ID= 'detector=0 detectorName=S00 filter=NONE calibDate=2018-09-21'        
            HIERARCH MD5_IMAGE = 'eda3f77dfa7ea894a0f796e24615a42a'                         
            MD5_MASK= 'fb394efc9596d6865b4dc311839da4f9'                                    
            HIERARCH MD5_VARIANCE = '8df0c2327f53ee09cb391743817c3f94'                      
            HIERARCH EXPINFO_V =         0                                                  
            AR_HDU  =                    5 / HDU (1-indexed) containing the archive used to 
            FILTER  = '_unknown_'                                                           
            FLUXMAG0=                   0.                                                  
            HIERARCH FLUXMAG0ERR =      0.                                                  
            END                             
            

            Show
            tjenness Tim Jenness added a comment - This was easier than I expected (to at least prototype). Changes needed to pipe_drivers and astro_metadata_translator but now the auxTel ci_lsst calibration output looks like: SIMPLE = T / file does conform to FITS standard BITPIX = 8 / number of bits per data pixel NAXIS = 0 / number of data axes EXTEND = T / FITS dataset may contain extensions COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H HEADVER = 2 / Version number of header INSTRUME= 'LSST_ATISS' / Instrument TELESCOP= 'LSST_AuxTel' / Telescope SEQFILE = 'ats_20180511.seq' / Sequencer file name CCD_MANU= 'ITL ' / CCD Manufacturer CCD_TYPE= '3800C ' / CCD Model Number CCD_SERN= '20304 ' / Manufacturers' CCD Serial Number LSST_NUM= 'ITL-3800C-098' / LSST Assigned CCD Number DETSIZE = '[1:4072,1:4000]' / IRAF detector size EXPTIME = 0. / Exposure Time in Seconds TELCODE = 'AT ' / The "code" for AuxTel CONTRLLR= 'C ' / The controller (e.g. O for OCS, C for CCS) DAYOBS = '20180920' / The observation day as defined in the image nam MJD-OBS = 58382.2355372338 / Modified Julian Date of image acquisition DATE-OBS= '2018-09-21T05:39:10.417' / Date of the observation (image acquisition DARKTIME= 0. ROTTYPE = 'UNKNOWN ' / Type of rotation angle OBSTYPE = 'bias ' HIERARCH CALIB_CREATION_DATE = '2019-02-15' HIERARCH CALIB_CREATION_TIME = '15:13:07 MST' DATE-AVG= '2018-09-21T00:00:00.00' HIERARCH CALIB_INPUT_0 = '(2018092000028,)' HIERARCH CALIB_INPUT_1 = '(2018092000029,)' HIERARCH CALIB_INPUT_2 = '(2018092000030,)' HIERARCH CALIB_INPUT_3 = '(2018092000031,)' HIERARCH CALIB_INPUT_4 = '(2018092000032,)' HIERARCH CALIB_INPUT_5 = '(2018092000033,)' HIERARCH CALIB_INPUT_6 = '(2018092000034,)' HIERARCH CALIB_INPUT_7 = '(2018092000035,)' HIERARCH CALIB_INPUT_8 = '(2018092000036,)' CALIB_ID= 'detector=0 detectorName=S00 filter=NONE calibDate=2018-09-21' HIERARCH MD5_IMAGE = 'eda3f77dfa7ea894a0f796e24615a42a' MD5_MASK= 'fb394efc9596d6865b4dc311839da4f9' HIERARCH MD5_VARIANCE = '8df0c2327f53ee09cb391743817c3f94' HIERARCH EXPINFO_V = 0 AR_HDU = 5 / HDU (1-indexed) containing the archive used to FILTER = '_unknown_' FLUXMAG0= 0. HIERARCH FLUXMAG0ERR = 0. END
            Hide
            rhl Robert Lupton added a comment -

            Putting extra information into the headers is fine, although we don't need it for LSST. The butler knows where to find the data, and the full list of input data used to generate the calibs is already in the metadata.

            I am more concerned about the DATE-OBS and DATE-END fields. These values don't tell us about the validity range of the calibs (that's set upon ingest into the calib registry), and these keys might imply the contrary.

            Show
            rhl Robert Lupton added a comment - Putting extra information into the headers is fine, although we don't need it for LSST. The butler knows where to find the data, and the full list of input data used to generate the calibs is already in the metadata. I am more concerned about the DATE-OBS and DATE-END fields. These values don't tell us about the validity range of the calibs (that's set upon ingest into the calib registry), and these keys might imply the contrary.
            tjenness Tim Jenness made changes -
            Link This issue is triggered by RFC-576 [ RFC-576 ]
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11 [ 862 ] Arch 2019-02-11, Arch 2019-02-18 [ 862, 863 ]
            swinbank John Swinbank made changes -
            Labels SciencePipelines
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18 [ 862, 863 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25 [ 862, 863, 867 ]
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25 [ 862, 863, 867 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04 [ 862, 863, 867, 869 ]
            Hide
            tjenness Tim Jenness added a comment -

            Following on from RFC-576, I've reworked the pull request to propagate headers without trying to forward first/last values of some special headers. For that I will have a go at reworking the VisitInfo logic in pipe_drivers so that I can get the date-start/end values from VisitInfo. I might fix the "average date" value as well since it seems wrong at the moment.

            Show
            tjenness Tim Jenness added a comment - Following on from RFC-576 , I've reworked the pull request to propagate headers without trying to forward first/last values of some special headers. For that I will have a go at reworking the VisitInfo logic in pipe_drivers so that I can get the date-start/end values from VisitInfo. I might fix the "average date" value as well since it seems wrong at the moment.
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04 [ 862, 863, 867, 869 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11 [ 862, 863, 867, 869, 870 ]
            Hide
            tjenness Tim Jenness added a comment -

            Paul Price would you be able to look at this? It's a small change to pipe_drivers constructCalibs.py to propagate input headers to outputs and to add some standard date headers. The metadata mangling itself is handed off to astro_metadata_translator.

            Show
            tjenness Tim Jenness added a comment - Paul Price would you be able to look at this? It's a small change to pipe_drivers constructCalibs.py to propagate input headers to outputs and to add some standard date headers. The metadata mangling itself is handed off to astro_metadata_translator.
            tjenness Tim Jenness made changes -
            Reviewers Paul Price [ price ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Story Points 2 4
            Hide
            price Paul Price added a comment -

            pipe_drivers changes look good. Please be sure to test them, as they do not currently get exercised by unit tests, the demo or ci_hsc. I think ci_ctio0m9 might do.

            Show
            price Paul Price added a comment - pipe_drivers changes look good. Please be sure to test them, as they do not currently get exercised by unit tests, the demo or ci_hsc. I think ci_ctio0m9 might do.
            Hide
            tjenness Tim Jenness added a comment -

            They work with ci_lsst (Which is where I test them). I will have a look at ci_ctio0m9.

            Show
            tjenness Tim Jenness added a comment - They work with ci_lsst (Which is where I test them). I will have a look at ci_ctio0m9.
            Hide
            tjenness Tim Jenness added a comment -

            Simon Krughoff thank you for agreeing to review the changes to astro_metadata_translator.

            Show
            tjenness Tim Jenness added a comment - Simon Krughoff thank you for agreeing to review the changes to astro_metadata_translator.
            tjenness Tim Jenness made changes -
            Reviewers Paul Price [ price ] Paul Price, Simon Krughoff [ price, krughoff ]
            Status In Review [ 10004 ] In Review [ 10004 ]
            Hide
            krughoff Simon Krughoff added a comment -

            As I understand it Paul Price is ok with the pipe_drivers changes. I've reviewed the other changes and have a few comments on the PR.

            Show
            krughoff Simon Krughoff added a comment - As I understand it Paul Price is ok with the pipe_drivers changes. I've reviewed the other changes and have a few comments on the PR.
            krughoff Simon Krughoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11 [ 862, 863, 867, 869, 870 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11, Arch 2019-03-18 [ 862, 863, 867, 869, 870, 871 ]
            Hide
            tjenness Tim Jenness added a comment -

            Thanks for the reviews. I tested with ci_ctio0m9 and tweaked the patch so that it functions properly if header translation fails. Jenkins passed and I tested again with ci_lsst. Merged.

            Show
            tjenness Tim Jenness added a comment - Thanks for the reviews. I tested with ci_ctio0m9 and tweaked the patch so that it functions properly if header translation fails. Jenkins passed and I tested again with ci_lsst. Merged.
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            tjenness Tim Jenness made changes -
            Epic Link DM-16576 [ 234914 ]

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Paul Price, Simon Krughoff
              Watchers:
              Jim Bosch, John Swinbank, Merlin Fisher-Levine, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.