Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Story Points:4
-
Team:Architecture
-
Urgent?:No
Description
It's clear we're missing a lot of standard, useful columns (many of them already enumerated in the ObservationInfo class) in the exposure, visit, and detector tables. We should try harder to get this list complete before we declare the Registry schema stable to reduce the number of migrations we have to do afterwards.
This includes:
- Target name (for ObsCore)
- RA/Dec/rotation angle/rotation coord (so that we can calculate visits without reading the raw data file)
Also includes changes to obs_base to use the ra/dec in visit region calculation.
Attachments
Issue Links
- blocks
-
DM-26039 Missing ObsCore fields for image metadata in Butler Gen3
- Done
- has to be finished together with
-
- Done
- is duplicated by
-
- Invalid
- is triggered by
-
- Implemented
- is triggering
-
- Done
Activity
No-one is asking for additional metadata so I may as well put this into review. If people want more they will have to do it on a separate ticket.
Regarding the define-visits performance. When I started this ticket the time it took to process 8 HSC visits was 19 seconds (total, including process start up and importing python code). Now by calculating the WCS using registry and by caching in DM-26317 it's 5 seconds – the actual visit region calculations are less than a second total.
The region calculation itself is fast enough that I now wonder if it's worth doing that as part of raw ingest (or at least having an exposure_detector_region table) – visit region then using exposure region directly. Granted calibration observations wouldn't want entries.
A few comments on the PR; mostly I think we should try to add these new ones to visit as well.
The region calculation itself is fast enough that I now wonder if it's worth doing that as part of raw ingest (or at least having an exposure_detector_region table) – visit region then using exposure region directly. Granted calibration observations wouldn't want entries.
I'd like to hold off on that until we have a use case, because it complicates the query logic that picks which regions to use to relate various dimensions. Nothing insurmountable, but a bit of a pain, at least.
I had added target name and science program to visit (it's in the obs_base PR) but I had failed to push my daf_butler branch. Updated now.
As for others: I can add zenith angle since that's in ObservationInfo. Is a straight average what people want to turn up in the visit table? cc/ Eli Rykoff?
ra/dec/sky angle seem tricky as well given that they are not guaranteed to be the same for all exposures in a visit (we have dithering and no guarantee the rotator hasn't moved).
Merged with the zenith angle addition.
I've updated the Gen 3 repository in ap_verify_testdata to match the new schema. That should be the only repository of ours that would affect Jenkins.