Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26039

Missing ObsCore fields for image metadata in Butler Gen3

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ImgServ
    • Labels:
    • Team:
      Data Access and Database

      Description

      The missing (or desired) fields are denoted by # FIXME

      in the generation script for ObsCore data for images:

      https://github.com/lsst/dax_imgserv/blob/tickets/DM-21433/integration/ci_hsc/gen_img_obscore.py

       

      Here're the FIXME line items:

            r["target_name"] = ? # FIXME: to be scheduler field ID 

            r["obs_id"] = ? # FIXME: to be online OBSID

            r["s_fov"] =  ?  # FIXME: fov - the diameter of a circle around s_ra, s_dec

       

      For reference on the above fields, see Gregory Dubois-Felsmann's notes: https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=~gpdf&title=Satisfying+ObsCore+from+the+Gen3+Butler+Schema

       

      As of now, the script succeeded in fetching all the other fields from the ci_hsc test dataset without loading the Exposure object of the corresponding FITS file into memory, taking about ~1 min (on my low-end PC) to process and generate 68 rows of ObsCore data. 

       

      FYI, the generated CVS raw output has been attached here as reference. This raw CVS output is to be combined with SQL DDL and DML scripts to insert the data into a local PostgreSQL database, which has been done with success, albeit using psql tool.

       

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Kenny Lo you've tagged this ticket as ImgServ and data access team and written it as if this is to implement fixes in ImgServ. What I'm going to do is implement the fix for target name on DM-24575 and leave this ticket for you for handling the change (and as we have discussed field-of-view is not directly a registry problem).

            Show
            tjenness Tim Jenness added a comment - Kenny Lo you've tagged this ticket as ImgServ and data access team and written it as if this is to implement fixes in ImgServ. What I'm going to do is implement the fix for target name on DM-24575 and leave this ticket for you for handling the change (and as we have discussed field-of-view is not directly a registry problem).
            Hide
            tjenness Tim Jenness added a comment -

            DM-24575 is now in review – I added science program and target name (as well as ra/dec).

            Show
            tjenness Tim Jenness added a comment - DM-24575 is now in review – I added science program and target name (as well as ra/dec).
            Hide
            tjenness Tim Jenness added a comment -

            Kenny Lo let me know if anything further needs to be done by me.

            Show
            tjenness Tim Jenness added a comment - Kenny Lo let me know if anything further needs to be done by me.
            Hide
            kennylo Kenny Lo added a comment -

            Tim Jenness FYI, I've been running/debugging the script on the HSC RC2 datasets on the verification cluster.  Will let you know if I run into anything.  

            Show
            kennylo Kenny Lo added a comment - Tim Jenness  FYI, I've been running/debugging the script on the HSC RC2 datasets on the verification cluster.  Will let you know if I run into anything.  
            Hide
            fritzm Fritz Mueller added a comment -

            Kenny has now transitioned off the DAX team.  Any further work beyond current state on this will be separately ticketed and staffed by the SQuaRE team.

            Show
            fritzm Fritz Mueller added a comment - Kenny has now transitioned off the DAX team.  Any further work beyond current state on this will be separately ticketed and staffed by the SQuaRE team.

              People

              Assignee:
              kennylo Kenny Lo
              Reporter:
              kennylo Kenny Lo
              Reviewers:
              Gregory Dubois-Felsmann, Kenny Lo
              Watchers:
              Fritz Mueller, Gregory Dubois-Felsmann, Kenny Lo, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.