Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34483

Butler Registry to ObsCore exporter

    XMLWordPrintable

Details

    • Ops Middleware
    • No

    Description

      The ObsTAP service requires that we have a database populated with ObsCore entries.

      We need something that can take a Registry (possibly with a dataset query constraint) and exports it to, say, a parquet file that can be ingested into the ObsCore database. Some values can come from configuration (such as the telescope location and how to map a dataset type to a processing level).

      Setting up the ObsCore database and loading these files into it can be done as a separate ticket.

      A proof-of-concept was written at https://github.com/lsst-dm/dax_obscore which attempts to show that this can be done. I don't know if any of this code can be used in the final system but it doesn't seem unreasonable to use the package itself dax_obscore.

      The code in dax_obscore has some problems in that it should never call butler.get – the sky region for an exposure must come from the visit record, as should the exposure time. calexp is visit based so that is fine, but for raw there might be a complication in getting the "footprint" in the absence of a direct link from an exposure record to a visit record.

      I envisage that we have a butler plugin click command that can do the equivalent of query-datasets and dump the results as a parquet table. Obviously we may also want a bespoke exporter that is distinct from the command-line tooling when doing a full export.

      gpdf how are you envisaging we use the visit concept in ObsCore? In theory we don't need it at all and can use the naive approach of every observation is an exposure. How many dataset types is the service meant to expose? raw, calexp and the tract/patch coadds?

      Attachments

        Issue Links

          Activity

            salnikov I have (finally) been looking closely at this, looking at what was released a while back (on 4/24).

            Do you have a CSV version of the most recent export format?

            gpdf Gregory Dubois-Felsmann added a comment - salnikov I have (finally) been looking closely at this, looking at what was released a while back (on 4/24). Do you have a CSV version of the most recent export format?

            gpdf, I have made a fresh copy of CSV from /repo/main using example.yaml configuration, it is at https://lsst.ncsa.illinois.edu/~salnikov/data.csv.gz, or in /project/salnikov/obscore/data.csv.gz on lsst-devl.

            salnikov Andy Salnikov added a comment - gpdf , I have made a fresh copy of CSV from /repo/main using example.yaml configuration, it is at https://lsst.ncsa.illinois.edu/~salnikov/data.csv.gz , or in /project/salnikov/obscore/data.csv.gz on lsst-devl.

            tjenness, sorry, I forgot to make PR for daf_butler earlier, it needed a small fix to avoid crashing: https://github.com/lsst/daf_butler/pull/682.

            salnikov Andy Salnikov added a comment - tjenness , sorry, I forgot to make PR for daf_butler earlier, it needed a small fix to avoid crashing: https://github.com/lsst/daf_butler/pull/682 .
            tjenness Tim Jenness added a comment -

            Looks great. Thanks for all the cleanups.

            tjenness Tim Jenness added a comment - Looks great. Thanks for all the cleanups.

            Thanks everyone for reviewing and all suggestions! Finally merged.

            salnikov Andy Salnikov added a comment - Thanks everyone for reviewing and all suggestions! Finally merged.

            People

              salnikov Andy Salnikov
              tjenness Tim Jenness
              Tim Jenness
              Andy Salnikov, Christine Banek, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.