Details
-
Type:
Story
-
Status: Reviewed
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: sdm_schemas
-
Labels:
-
Team:Architecture
-
Urgent?:No
Description
Following the completion of Andy Salnikov's initial work on dax_obscore to create the basic Butler-to-ObsCore metadata extraction capability, create the production config file for the DP0.2 data extraction in a new /obscore subdirectory of sdm_schemas.
Final fine-tuning of the configuration can/should happen there, rather than on the sample config files in dax_obscore.
Attachments
Issue Links
Activity
Colin asked me to post to this ticket so that the issue isn't forgotten, but please redirect as needed.
I was exploring the TAP image access services and (re-)discovered that the units on `em_min` and `em_max` are meters, while the loaded values are in nanometers. This was first noticed by Tim [here|https://lsstc.slack.com/archives/C8EEUGDSA/p1653691283701969?thread_ts=1653690826.504649&cid=C8EEUGDSA]. I have confirmed that queries do work, but it means you need to do some whacky conversion since the Portal only provides microns, nanometers, and angstroms.
I've looked at the config that was used on the initial test run (https://github.com/lsst-dm/dax_obscore/blob/main/configs/dp02.yaml), and the set of images looks correct. The new collection to extract from is "2.2i/runs/DP0.2". And the right fix for the behavior Alex reported is to change the spectral_ranges entries to be in meters, e.g. "u": [330.0e-9, 400.0e-9]
Colin Slater
added a comment - I've looked at the config that was used on the initial test run ( https://github.com/lsst-dm/dax_obscore/blob/main/configs/dp02.yaml ), and the set of images looks correct. The new collection to extract from is "2.2i/runs/DP0.2". And the right fix for the behavior Alex reported is to change the spectral_ranges entries to be in meters, e.g. "u": [330.0e-9, 400.0e-9] Thanks, Colin! We'll wait for DM-35301, then I'll put up a PR with these changes as well as the new (lsst_) visit, detector, tract, patch, band, filter columns.
Changes per commentary. Running the extraction right now with config from this branch...
Okay, I got:
[fritzm@nb-fritzm dax_obscore]$ butler obscore-export -c configs/dp02.yaml --format csv dp02 obscore.csv |
WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01). |
WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01). |
WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01). |
lsst.dax.obscore.obscore_exporter INFO: Copied 2870807 records from dataset type raw |
lsst.dax.obscore.obscore_exporter INFO: Copied 2805017 records from dataset type calexp |
lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type deepCoadd_calexp |
lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type goodSeeingCoadd |
lsst.dax.obscore.obscore_exporter INFO: Copied 2707834 records from dataset type goodSeeingDiff_differenceExp |
[fritzm@nb-fritzm dax_obscore]$ wc -l obscore.csv |
8475975 obscore.csv |
I can't remember if there was something this review was waiting on, but it's certainly done now.
Colin Slater
added a comment - I can't remember if there was something this review was waiting on, but it's certainly done now.
Is this really a SQuaRE ticket or should it be a DM Science ticket?