Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16292

Calibration products need more metadata

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_drivers
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11, Arch 2019-03-18
    • Team:
      Architecture

      Description

      I was testing the new metadata translator on FITS file I had lying around in testdata repositories and I see that our calibration products do not seem to have a reasonable set of headers. The only indication this is a HSC calibration file rather than an LSST one is the path name to the root directory.

      SIMPLE  =                    T / file does conform to FITS standard             
      BITPIX  =                  -32 / number of bits per data pixel                  
      NAXIS   =                    2 / number of data axes                            
      NAXIS1  =                 2048 / length of data axis 1                          
      NAXIS2  =                 4176 / length of data axis 2                          
      EXTEND  =                    T / FITS dataset may contain extensions            
      COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
      COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
      OBSTYPE = 'bias    '                                                            
      HIERARCH CALIB_CREATION_DATE = '2016-03-28'                                     
      HIERARCH CALIB_CREATION_TIME = '16:16:33 EDT'                                   
      HIERARCH CALIB_CREATION_ROOT = '/tigress/HSC/HSC/rerun/dm-5124/calib'           
      HIERARCH CALIB_INPUT_0 = '(904542,)'                                            
      HIERARCH CALIB_INPUT_1 = '(904544,)'                                            
      HIERARCH CALIB_INPUT_2 = '(904546,)'                                            
      HIERARCH CALIB_INPUT_3 = '(904548,)'                                            
      HIERARCH CALIB_INPUT_4 = '(904550,)'                                            
      HIERARCH CALIB_INPUT_5 = '(904552,)'                                            
      HIERARCH CALIB_INPUT_6 = '(904554,)'                                            
      HIERARCH CALIB_INPUT_7 = '(904556,)'                                            
      HIERARCH CALIB_INPUT_8 = '(904558,)'                                            
      HIERARCH CALIB_INPUT_9 = '(904560,)'                                            
      HIERARCH CALIB_INPUT_10 = '(904562,)'                                           
      HIERARCH CALIB_INPUT_11 = '(904564,)'                                           
      HIERARCH CALIB_INPUT_12 = '(904566,)'                                           
      CALIB_ID= 'filter=NONE calibDate=2013-11-03 ccd=50'                             
      HIERARCH MD5_IMAGE = '6185bb72f20de7e81c45e6e6591eb6ad'                         
      CRVAL1A =                    0 / Column pixel of Reference Pixel                
      CRVAL2A =                    0 / Row pixel of Reference Pixel                   
      CRPIX1A =                    1 / Column Pixel Coordinate of Reference           
      CRPIX2A =                    1 / Row Pixel Coordinate of Reference              
      CTYPE1A = 'LINEAR  '           / Type of projection                             
      CTYPE2A = 'LINEAR  '           / Type of projection                             
      CUNIT1A = 'PIXEL   '           / Column unit                                    
      CUNIT2A = 'PIXEL   '           / Row unit                                      
      

      At minimum there should be INSTRUME/TELESCOP headers but ideally we should be able to construct a full header based on all the input headers (propagating headers that are identical and indicating which headers define bounds). The latter is what I implemented at the JCMT Science Archive for all data products. It might be that another approach is to at least be able to generate a standard set of headers from the information the Butler (Gen 3) knows about each dataset.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            pipe_drivers changes look good. Please be sure to test them, as they do not currently get exercised by unit tests, the demo or ci_hsc. I think ci_ctio0m9 might do.

            Show
            price Paul Price added a comment - pipe_drivers changes look good. Please be sure to test them, as they do not currently get exercised by unit tests, the demo or ci_hsc. I think ci_ctio0m9 might do.
            Hide
            tjenness Tim Jenness added a comment -

            They work with ci_lsst (Which is where I test them). I will have a look at ci_ctio0m9.

            Show
            tjenness Tim Jenness added a comment - They work with ci_lsst (Which is where I test them). I will have a look at ci_ctio0m9.
            Hide
            tjenness Tim Jenness added a comment -

            Simon Krughoff thank you for agreeing to review the changes to astro_metadata_translator.

            Show
            tjenness Tim Jenness added a comment - Simon Krughoff thank you for agreeing to review the changes to astro_metadata_translator.
            Hide
            krughoff Simon Krughoff added a comment -

            As I understand it Paul Price is ok with the pipe_drivers changes. I've reviewed the other changes and have a few comments on the PR.

            Show
            krughoff Simon Krughoff added a comment - As I understand it Paul Price is ok with the pipe_drivers changes. I've reviewed the other changes and have a few comments on the PR.
            Hide
            tjenness Tim Jenness added a comment -

            Thanks for the reviews. I tested with ci_ctio0m9 and tweaked the patch so that it functions properly if header translation fails. Jenkins passed and I tested again with ci_lsst. Merged.

            Show
            tjenness Tim Jenness added a comment - Thanks for the reviews. I tested with ci_ctio0m9 and tweaked the patch so that it functions properly if header translation fails. Jenkins passed and I tested again with ci_lsst. Merged.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                Paul Price, Simon Krughoff
                Watchers:
                Jim Bosch, John Swinbank, Merlin Fisher-Levine, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel