Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16292

Calibration products need more metadata

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_drivers
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11, Arch 2019-03-18
    • Team:
      Architecture

      Description

      I was testing the new metadata translator on FITS file I had lying around in testdata repositories and I see that our calibration products do not seem to have a reasonable set of headers. The only indication this is a HSC calibration file rather than an LSST one is the path name to the root directory.

      SIMPLE  =                    T / file does conform to FITS standard             
      BITPIX  =                  -32 / number of bits per data pixel                  
      NAXIS   =                    2 / number of data axes                            
      NAXIS1  =                 2048 / length of data axis 1                          
      NAXIS2  =                 4176 / length of data axis 2                          
      EXTEND  =                    T / FITS dataset may contain extensions            
      COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
      COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
      OBSTYPE = 'bias    '                                                            
      HIERARCH CALIB_CREATION_DATE = '2016-03-28'                                     
      HIERARCH CALIB_CREATION_TIME = '16:16:33 EDT'                                   
      HIERARCH CALIB_CREATION_ROOT = '/tigress/HSC/HSC/rerun/dm-5124/calib'           
      HIERARCH CALIB_INPUT_0 = '(904542,)'                                            
      HIERARCH CALIB_INPUT_1 = '(904544,)'                                            
      HIERARCH CALIB_INPUT_2 = '(904546,)'                                            
      HIERARCH CALIB_INPUT_3 = '(904548,)'                                            
      HIERARCH CALIB_INPUT_4 = '(904550,)'                                            
      HIERARCH CALIB_INPUT_5 = '(904552,)'                                            
      HIERARCH CALIB_INPUT_6 = '(904554,)'                                            
      HIERARCH CALIB_INPUT_7 = '(904556,)'                                            
      HIERARCH CALIB_INPUT_8 = '(904558,)'                                            
      HIERARCH CALIB_INPUT_9 = '(904560,)'                                            
      HIERARCH CALIB_INPUT_10 = '(904562,)'                                           
      HIERARCH CALIB_INPUT_11 = '(904564,)'                                           
      HIERARCH CALIB_INPUT_12 = '(904566,)'                                           
      CALIB_ID= 'filter=NONE calibDate=2013-11-03 ccd=50'                             
      HIERARCH MD5_IMAGE = '6185bb72f20de7e81c45e6e6591eb6ad'                         
      CRVAL1A =                    0 / Column pixel of Reference Pixel                
      CRVAL2A =                    0 / Row pixel of Reference Pixel                   
      CRPIX1A =                    1 / Column Pixel Coordinate of Reference           
      CRPIX2A =                    1 / Row Pixel Coordinate of Reference              
      CTYPE1A = 'LINEAR  '           / Type of projection                             
      CTYPE2A = 'LINEAR  '           / Type of projection                             
      CUNIT1A = 'PIXEL   '           / Column unit                                    
      CUNIT2A = 'PIXEL   '           / Row unit                                      
      

      At minimum there should be INSTRUME/TELESCOP headers but ideally we should be able to construct a full header based on all the input headers (propagating headers that are identical and indicating which headers define bounds). The latter is what I implemented at the JCMT Science Archive for all data products. It might be that another approach is to at least be able to generate a standard set of headers from the information the Butler (Gen 3) knows about each dataset.

        Attachments

          Issue Links

            Activity

            tjenness Tim Jenness created issue -
            swinbank John Swinbank made changes -
            Field Original Value New Value
            Remote Link This issue links to "Page (Confluence)" [ 19479 ]
            tjenness Tim Jenness made changes -
            Component/s pipe_drivers [ 13309 ]
            tjenness Tim Jenness made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Assignee Tim Jenness [ tjenness ]
            tjenness Tim Jenness made changes -
            Sprint Arch 2019-02-11 [ 862 ]
            Story Points 2
            Team Architecture [ 10304 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggered by RFC-576 [ RFC-576 ]
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11 [ 862 ] Arch 2019-02-11, Arch 2019-02-18 [ 862, 863 ]
            swinbank John Swinbank made changes -
            Labels SciencePipelines
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18 [ 862, 863 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25 [ 862, 863, 867 ]
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25 [ 862, 863, 867 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04 [ 862, 863, 867, 869 ]
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04 [ 862, 863, 867, 869 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11 [ 862, 863, 867, 869, 870 ]
            tjenness Tim Jenness made changes -
            Reviewers Paul Price [ price ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Story Points 2 4
            tjenness Tim Jenness made changes -
            Reviewers Paul Price [ price ] Paul Price, Simon Krughoff [ price, krughoff ]
            Status In Review [ 10004 ] In Review [ 10004 ]
            krughoff Simon Krughoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            ktl Kian-Tat Lim made changes -
            Sprint Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11 [ 862, 863, 867, 869, 870 ] Arch 2019-02-11, Arch 2019-02-18, Arch 2019-02-25, Arch 2019-03-04, Arch 2019-03-11, Arch 2019-03-18 [ 862, 863, 867, 869, 870, 871 ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            tjenness Tim Jenness made changes -
            Epic Link DM-16576 [ 234914 ]

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Paul Price, Simon Krughoff
              Watchers:
              Jim Bosch, John Swinbank, Merlin Fisher-Levine, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.