# Create production Butler-to-ObsCore configuration file(s) in sdm_schemas/obscore

XMLWordPrintable

#### Details

• Type: Story
• Status: Reviewed
• Resolution: Unresolved
• Fix Version/s: None
• Component/s:
• Labels:
• Team:
Architecture
• Urgent?:
No

#### Description

Following the completion of Andy Salnikov's initial work on dax_obscore to create the basic Butler-to-ObsCore metadata extraction capability, create the production config file for the DP0.2 data extraction in a new /obscore subdirectory of sdm_schemas.

Final fine-tuning of the configuration can/should happen there, rather than on the sample config files in dax_obscore.

#### Attachments

1. portal_image.png
18 kB

#### Activity

Hide
Tim Jenness added a comment -

Is this really a SQuaRE ticket or should it be a DM Science ticket?

Show
Tim Jenness added a comment - Is this really a SQuaRE ticket or should it be a DM Science ticket?
Hide
Alex Drlica-Wagner added a comment -

Colin asked me to post to this ticket so that the issue isn't forgotten, but please redirect as needed.

I was exploring the TAP image access services and (re-)discovered that the units on em_min and em_max are meters, while the loaded values are in nanometers. This was first noticed by Tim [here|https://lsstc.slack.com/archives/C8EEUGDSA/p1653691283701969?thread_ts=1653690826.504649&cid=C8EEUGDSA]. I have confirmed that queries do work, but it means you need to do some whacky conversion since the Portal only provides microns, nanometers, and angstroms.

Show
Alex Drlica-Wagner added a comment - Colin asked me to post to this ticket so that the issue isn't forgotten, but please redirect as needed. I was exploring the TAP image access services and (re-)discovered that the units on em_min and em_max are meters, while the loaded values are in nanometers. This was first noticed by Tim [here| https://lsstc.slack.com/archives/C8EEUGDSA/p1653691283701969?thread_ts=1653690826.504649&cid=C8EEUGDSA ]. I have confirmed that queries do work, but it means you need to do some whacky conversion since the Portal only provides microns, nanometers, and angstroms.
Hide
Colin Slater added a comment -

I've looked at the config that was used on the initial test run (https://github.com/lsst-dm/dax_obscore/blob/main/configs/dp02.yaml), and the set of images looks correct. The new collection to extract from is "2.2i/runs/DP0.2". And the right fix for the behavior Alex reported is to change the spectral_ranges entries to be in meters, e.g. "u": [330.0e-9, 400.0e-9]

Show
Colin Slater added a comment - I've looked at the config that was used on the initial test run ( https://github.com/lsst-dm/dax_obscore/blob/main/configs/dp02.yaml ), and the set of images looks correct. The new collection to extract from is "2.2i/runs/DP0.2". And the right fix for the behavior Alex reported is to change the spectral_ranges entries to be in meters, e.g. "u": [330.0e-9, 400.0e-9]
Hide
Fritz Mueller added a comment -

Thanks, Colin!  We'll wait for DM-35301, then I'll put up a PR with these changes as well as the new (lsst_) visit, detector, tract, patch, band, filter columns.

Show
Fritz Mueller added a comment - Thanks, Colin!  We'll wait for DM-35301 , then I'll put up a PR with these changes as well as the new (lsst_) visit, detector, tract, patch, band, filter columns.
Hide
Fritz Mueller added a comment -

Changes per commentary.  Running the extraction right now with config from this branch...

Show
Fritz Mueller added a comment - Changes per commentary.  Running the extraction right now with config from this branch...
Hide
Fritz Mueller added a comment -

Okay, I got:

 [fritzm@nb-fritzm dax_obscore]$butler obscore-export -c configs/dp02.yaml --format csv dp02 obscore.csv WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01). WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01). WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01). lsst.dax.obscore.obscore_exporter INFO: Copied 2870807 records from dataset type raw lsst.dax.obscore.obscore_exporter INFO: Copied 2805017 records from dataset type calexp lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type deepCoadd_calexp lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type goodSeeingCoadd lsst.dax.obscore.obscore_exporter INFO: Copied 2707834 records from dataset type goodSeeingDiff_differenceExp [fritzm@nb-fritzm dax_obscore]$ wc -l obscore.csv  8475975 obscore.csv 

Show
Fritz Mueller added a comment - Okay, I got:  [fritzm @nb -fritzm dax_obscore]$butler obscore-export -c configs/dp02.yaml --format csv dp02 obscore.csv WARNING: version mismatch between CFITSIO header (v4. 000999999999999 ) and linked library (v4. 01 ). WARNING: version mismatch between CFITSIO header (v4. 000999999999999 ) and linked library (v4. 01 ). WARNING: version mismatch between CFITSIO header (v4. 000999999999999 ) and linked library (v4. 01 ). lsst.dax.obscore.obscore_exporter INFO: Copied 2870807 records from dataset type raw lsst.dax.obscore.obscore_exporter INFO: Copied 2805017 records from dataset type calexp lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type deepCoadd_calexp lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type goodSeeingCoadd lsst.dax.obscore.obscore_exporter INFO: Copied 2707834 records from dataset type goodSeeingDiff_differenceExp [fritzm @nb -fritzm dax_obscore]$ wc -l obscore.csv 8475975 obscore.csv
Hide
Colin Slater added a comment -

I can't remember if there was something this review was waiting on, but it's certainly done now.

Show
Colin Slater added a comment - I can't remember if there was something this review was waiting on, but it's certainly done now.

#### People

Assignee:
Fritz Mueller
Reporter:
Gregory Dubois-Felsmann
Reviewers:
Colin Slater
Watchers:
Alex Drlica-Wagner, Andy Salnikov, Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Tim Jenness