Details
Description
The missing (or desired) fields are denoted by # FIXME
in the generation script for ObsCore data for images:
https://github.com/lsst/dax_imgserv/blob/tickets/DM-21433/integration/ci_hsc/gen_img_obscore.py
Here're the FIXME line items:
r["target_name"] = ? # FIXME: to be scheduler field ID
r["obs_id"] = ? # FIXME: to be online OBSID
r["s_fov"] = ? # FIXME: fov - the diameter of a circle around s_ra, s_dec
For reference on the above fields, see Gregory Dubois-Felsmann's notes: https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=~gpdf&title=Satisfying+ObsCore+from+the+Gen3+Butler+Schema
As of now, the script succeeded in fetching all the other fields from the ci_hsc test dataset without loading the Exposure object of the corresponding FITS file into memory, taking about ~1 min (on my low-end PC) to process and generate 68 rows of ObsCore data.
FYI, the generated CVS raw output has been attached here as reference. This raw CVS output is to be combined with SQL DDL and DML scripts to insert the data into a local PostgreSQL database, which has been done with success, albeit using psql tool.
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
The missing (or desired) fields are denoted by # FIXME
in the generation script for ObsCore data for images: [https://github.com/lsst/dax_imgserv/blob/tickets/DM-21433/integration/ci_hsc/gen_img_obscore.py] As of now, the script succeeded in fetching all the other fields from the ci_hsc test dataset without loading the Exposure object of the corresponding FITS file into memory, taking about ~1 min (on my low-end PC) to process and generate 68 rows of ObsCore data. FYI, the generated CVS raw output has been attached here for reference. This raw CVS output is to be combined with SQL DDL script to insert the data into a local PostgreSQL database, which has been done with success, albeit using psql tool. |
The missing (or desired) fields are denoted by # FIXME
in the generation script for ObsCore data for images: [https://github.com/lsst/dax_imgserv/blob/tickets/DM-21433/integration/ci_hsc/gen_img_obscore.py] Here're the FIXME line items: r["target_name"] = ? # FIXME: to be scheduler field ID r["obs_id"] = ? # FIXME: to be online OBSID r["s_fov"] = ? # FIXME: fov - the diameter of a circle around s_ra, s_dec For reference on the above fields, see [~gpdf]'s notes: [https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=~gpdf&title=Satisfying+ObsCore+from+the+Gen3+Butler+Schema] As of now, the script succeeded in fetching all the other fields from the ci_hsc test dataset without loading the Exposure object of the corresponding FITS file into memory, taking about ~1 min (on my low-end PC) to process and generate 68 rows of ObsCore data. FYI, the generated CVS raw output has been attached here as reference. This raw CVS output is to be combined with SQL DDL script to insert the data into a local PostgreSQL database, which has been done with success, albeit using psql tool. |
| Description |
The missing (or desired) fields are denoted by # FIXME
in the generation script for ObsCore data for images: [https://github.com/lsst/dax_imgserv/blob/tickets/DM-21433/integration/ci_hsc/gen_img_obscore.py] Here're the FIXME line items: r["target_name"] = ? # FIXME: to be scheduler field ID r["obs_id"] = ? # FIXME: to be online OBSID r["s_fov"] = ? # FIXME: fov - the diameter of a circle around s_ra, s_dec For reference on the above fields, see [~gpdf]'s notes: [https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=~gpdf&title=Satisfying+ObsCore+from+the+Gen3+Butler+Schema] As of now, the script succeeded in fetching all the other fields from the ci_hsc test dataset without loading the Exposure object of the corresponding FITS file into memory, taking about ~1 min (on my low-end PC) to process and generate 68 rows of ObsCore data. FYI, the generated CVS raw output has been attached here as reference. This raw CVS output is to be combined with SQL DDL script to insert the data into a local PostgreSQL database, which has been done with success, albeit using psql tool. |
The missing (or desired) fields are denoted by # FIXME
in the generation script for ObsCore data for images: [https://github.com/lsst/dax_imgserv/blob/tickets/DM-21433/integration/ci_hsc/gen_img_obscore.py] Here're the FIXME line items: r["target_name"] = ? # FIXME: to be scheduler field ID r["obs_id"] = ? # FIXME: to be online OBSID r["s_fov"] = ? # FIXME: fov - the diameter of a circle around s_ra, s_dec For reference on the above fields, see [~gpdf]'s notes: [https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=~gpdf&title=Satisfying+ObsCore+from+the+Gen3+Butler+Schema] As of now, the script succeeded in fetching all the other fields from the ci_hsc test dataset without loading the Exposure object of the corresponding FITS file into memory, taking about ~1 min (on my low-end PC) to process and generate 68 rows of ObsCore data. FYI, the generated CVS raw output has been attached here as reference. This raw CVS output is to be combined with SQL DDL and DML scripts to insert the data into a local PostgreSQL database, which has been done with success, albeit using psql tool. |
s_fov will differ for single-epoch (CCD) images vs. coadd patches. I think the lookup should be by instrument AND Butler dataset type.
If target_name is going to be based on the scheduler field ID (for single-epoch images), it should have a prefix that defines a namespace, e.g., "Sched_00307", to allow these to be distinguished from other possible targets - for instance, it could be a special ID for TOO observations ("TOO_00017") and for calibration frames it could be "DomeScreen" or some such.
I don't know if we have to understand all of that now, though? Just make sure there's a prefix.
All I have is the OBJECT header. You get whatever it is they put in there.
What goes in there is an upstream issue.
Kenny Lo you've tagged this ticket as ImgServ and data access team and written it as if this is to implement fixes in ImgServ. What I'm going to do is implement the fix for target name on DM-24575 and leave this ticket for you for handling the change (and as we have discussed field-of-view is not directly a registry problem).
| Assignee | Tim Jenness [ tjenness ] | Kenny Lo [ kennylo ] |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
DM-24575 is now in review – I added science program and target name (as well as ra/dec).
| Labels | ObsCore |
Kenny Lo let me know if anything further needs to be done by me.
Tim Jenness FYI, I've been running/debugging the script on the HSC RC2 datasets on the verification cluster. Will let you know if I run into anything.
| Resolution | Done [ 10000 ] | |
| Status | In Progress [ 3 ] | Done [ 10002 ] |
Kenny has now transitioned off the DAX team. Any further work beyond current state on this will be separately ticketed and staffed by the SQuaRE team.
I may be confused but obs_id for an exposure is the name of the exposure in the gen3 record. You can see this by looking at https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/_instrument.py#L499 – it's the unique string identifying the observation (from the OBSID header).
The target name I can add (I already calculate it in ObservationInfo).
Field of view seems to be a slightly different topic in that from your point of view it's fixed for the camera. Butler calculates the region for visits and puts them in the visit table but it's never going to change for LSSTCam or ComCam so you can have a lookup by instrument name and use a constant.