Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34685

Create production Butler-to-ObsCore configuration file(s) in sdm_schemas/obscore

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Reviewed
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: sdm_schemas
    • Labels:
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Following the completion of Andy Salnikov's initial work on dax_obscore to create the basic Butler-to-ObsCore metadata extraction capability, create the production config file for the DP0.2 data extraction in a new /obscore subdirectory of sdm_schemas.

      Final fine-tuning of the configuration can/should happen there, rather than on the sample config files in dax_obscore.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35301 Extend dax_obscore extra_columns facility to support value templating

          • Done
          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34483 Butler Registry to ObsCore exporter

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35325 ObsCore extractor config fixes for DP0.2

          • To Do

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            gpdf Gregory Dubois-Felsmann created issue -
            gpdf Gregory Dubois-Felsmann made changes -
            Field Original Value New Value
            Link This issue is triggered by DM-34483 [ DM-34483 ]
            frossie Frossie Economou made changes -
            Assignee Frossie Economou [ frossie ]
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Is this really a SQuaRE ticket or should it be a DM Science ticket?

            Show
            tjenness Tim Jenness added a comment - Is this really a SQuaRE ticket or should it be a DM Science ticket?
            tjenness Tim Jenness made changes -
            Assignee Frossie Economou [ frossie ] Gregory Dubois-Felsmann [ gpdf ]
            tjenness Tim Jenness made changes -
            Team Architecture [ 10304 ]
            kadrlica Alex Drlica-Wagner made changes -
            Attachment portal_image.png [ 60736 ]
            Hide
            Permalink
            kadrlica Alex Drlica-Wagner added a comment -

            Colin asked me to post to this ticket so that the issue isn't forgotten, but please redirect as needed.

            I was exploring the TAP image access services and (re-)discovered that the units on `em_min` and `em_max` are meters, while the loaded values are in nanometers. This was first noticed by Tim [here|https://lsstc.slack.com/archives/C8EEUGDSA/p1653691283701969?thread_ts=1653690826.504649&cid=C8EEUGDSA]. I have confirmed that queries do work, but it means you need to do some whacky conversion since the Portal only provides microns, nanometers, and angstroms.

            Show
            kadrlica Alex Drlica-Wagner added a comment - Colin asked me to post to this ticket so that the issue isn't forgotten, but please redirect as needed. I was exploring the TAP image access services and (re-)discovered that the units on `em_min` and `em_max` are meters, while the loaded values are in nanometers. This was first noticed by Tim [here| https://lsstc.slack.com/archives/C8EEUGDSA/p1653691283701969?thread_ts=1653690826.504649&cid=C8EEUGDSA ]. I have confirmed that queries do work, but it means you need to do some whacky conversion since the Portal only provides microns, nanometers, and angstroms.
            fritzm Fritz Mueller made changes -
            Link This issue is blocked by DM-35301 [ DM-35301 ]
            Hide
            Permalink
            ctslater Colin Slater added a comment -

            I've looked at the config that was used on the initial test run (https://github.com/lsst-dm/dax_obscore/blob/main/configs/dp02.yaml), and the set of images looks correct. The new collection to extract from is "2.2i/runs/DP0.2". And the right fix for the behavior Alex reported is to change the spectral_ranges entries to be in meters, e.g. "u": [330.0e-9, 400.0e-9]

            Show
            ctslater Colin Slater added a comment - I've looked at the config that was used on the initial test run ( https://github.com/lsst-dm/dax_obscore/blob/main/configs/dp02.yaml ), and the set of images looks correct. The new collection to extract from is "2.2i/runs/DP0.2". And the right fix for the behavior Alex reported is to change the spectral_ranges entries to be in meters, e.g. "u": [330.0e-9, 400.0e-9]
            Hide
            Permalink
            fritzm Fritz Mueller added a comment -

            Thanks, Colin!  We'll wait for DM-35301, then I'll put up a PR with these changes as well as the new (lsst_) visit, detector, tract, patch, band, filter columns.

            Show
            fritzm Fritz Mueller added a comment - Thanks, Colin!  We'll wait for DM-35301 , then I'll put up a PR with these changes as well as the new (lsst_) visit, detector, tract, patch, band, filter columns.
            fritzm Fritz Mueller made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            fritzm Fritz Mueller made changes -
            Assignee Gregory Dubois-Felsmann [ gpdf ] Fritz Mueller [ fritzm ]
            fritzm Fritz Mueller made changes -
            Reviewers Gregory Dubois-Felsmann [ gpdf ] Colin Slater, Gregory Dubois-Felsmann [ ctslater, gpdf ]
            fritzm Fritz Mueller made changes -
            Reviewers Colin Slater, Gregory Dubois-Felsmann [ ctslater, gpdf ] Colin Slater [ ctslater ]
            Hide
            Permalink
            fritzm Fritz Mueller added a comment -

            Changes per commentary.  Running the extraction right now with config from this branch...

            Show
            fritzm Fritz Mueller added a comment - Changes per commentary.  Running the extraction right now with config from this branch...
            fritzm Fritz Mueller made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            fritzm Fritz Mueller added a comment -

            Okay, I got: 

            [fritzm@nb-fritzm dax_obscore]$ butler obscore-export -c configs/dp02.yaml --format csv dp02 obscore.csv
            		 
            WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01).
            		 
            WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01).
            		 
            WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01).
            		 
            lsst.dax.obscore.obscore_exporter INFO: Copied 2870807 records from dataset type raw
            		 
            lsst.dax.obscore.obscore_exporter INFO: Copied 2805017 records from dataset type calexp
            		 
            lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type deepCoadd_calexp
            		 
            lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type goodSeeingCoadd
            		 
            lsst.dax.obscore.obscore_exporter INFO: Copied 2707834 records from dataset type goodSeeingDiff_differenceExp
            		 
            [fritzm@nb-fritzm dax_obscore]$ wc -l obscore.csv 
            8475975 obscore.csv

             

            Show
            fritzm Fritz Mueller added a comment - Okay, I got:  [fritzm @nb -fritzm dax_obscore]$ butler obscore-export -c configs/dp02.yaml --format csv dp02 obscore.csv WARNING: version mismatch between CFITSIO header (v4. 000999999999999 ) and linked library (v4. 01 ). WARNING: version mismatch between CFITSIO header (v4. 000999999999999 ) and linked library (v4. 01 ). WARNING: version mismatch between CFITSIO header (v4. 000999999999999 ) and linked library (v4. 01 ). lsst.dax.obscore.obscore_exporter INFO: Copied 2870807 records from dataset type raw lsst.dax.obscore.obscore_exporter INFO: Copied 2805017 records from dataset type calexp lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type deepCoadd_calexp lsst.dax.obscore.obscore_exporter INFO: Copied 46158 records from dataset type goodSeeingCoadd lsst.dax.obscore.obscore_exporter INFO: Copied 2707834 records from dataset type goodSeeingDiff_differenceExp [fritzm @nb -fritzm dax_obscore]$ wc -l obscore.csv 8475975 obscore.csv  
            fritzm Fritz Mueller made changes -
            Link This issue relates to DM-35325 [ DM-35325 ]
            salnikov Andy Salnikov made changes -
            Labels ObsCore
            Hide
            Permalink
            ctslater Colin Slater added a comment -

            I can't remember if there was something this review was waiting on, but it's certainly done now.

            Show
            ctslater Colin Slater added a comment - I can't remember if there was something this review was waiting on, but it's certainly done now.
            ctslater Colin Slater made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]

              People

              Assignee:
              fritzm Fritz Mueller
              Reporter:
              gpdf Gregory Dubois-Felsmann
              Reviewers:
              Colin Slater
              Watchers:
              Alex Drlica-Wagner, Andy Salnikov, Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Tim Jenness
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.