Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34483

Butler Registry to ObsCore exporter

    XMLWordPrintable

    Details

    • Team:
      Ops Middleware
    • Urgent?:
      No

      Description

      The ObsTAP service requires that we have a database populated with ObsCore entries.

      We need something that can take a Registry (possibly with a dataset query constraint) and exports it to, say, a parquet file that can be ingested into the ObsCore database. Some values can come from configuration (such as the telescope location and how to map a dataset type to a processing level).

      Setting up the ObsCore database and loading these files into it can be done as a separate ticket.

      A proof-of-concept was written at https://github.com/lsst-dm/dax_obscore which attempts to show that this can be done. I don't know if any of this code can be used in the final system but it doesn't seem unreasonable to use the package itself dax_obscore.

      The code in dax_obscore has some problems in that it should never call butler.get – the sky region for an exposure must come from the visit record, as should the exposure time. calexp is visit based so that is fine, but for raw there might be a complication in getting the "footprint" in the absence of a direct link from an exposure record to a visit record.

      I envisage that we have a butler plugin click command that can do the equivalent of query-datasets and dump the results as a parquet table. Obviously we may also want a bespoke exporter that is distinct from the command-line tooling when doing a full export.

      Gregory Dubois-Felsmann how are you envisaging we use the visit concept in ObsCore? In theory we don't need it at all and can use the naive approach of every observation is an exposure. How many dataset types is the service meant to expose? raw, calexp and the tract/patch coadds?

        Attachments

          Issue Links

            Activity

            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            Andy Salnikov I have (finally) been looking closely at this, looking at what was released a while back (on 4/24).

            Do you have a CSV version of the most recent export format?

            Show
            gpdf Gregory Dubois-Felsmann added a comment - Andy Salnikov I have (finally) been looking closely at this, looking at what was released a while back (on 4/24). Do you have a CSV version of the most recent export format?
            Hide
            salnikov Andy Salnikov added a comment -

            Gregory Dubois-Felsmann, I have made a fresh copy of CSV from /repo/main using example.yaml configuration, it is at https://lsst.ncsa.illinois.edu/~salnikov/data.csv.gz, or in /project/salnikov/obscore/data.csv.gz on lsst-devl.

            Show
            salnikov Andy Salnikov added a comment - Gregory Dubois-Felsmann , I have made a fresh copy of CSV from /repo/main using example.yaml configuration, it is at https://lsst.ncsa.illinois.edu/~salnikov/data.csv.gz , or in /project/salnikov/obscore/data.csv.gz on lsst-devl.
            Hide
            salnikov Andy Salnikov added a comment -

            Tim Jenness, sorry, I forgot to make PR for daf_butler earlier, it needed a small fix to avoid crashing: https://github.com/lsst/daf_butler/pull/682.

            Show
            salnikov Andy Salnikov added a comment - Tim Jenness , sorry, I forgot to make PR for daf_butler earlier, it needed a small fix to avoid crashing: https://github.com/lsst/daf_butler/pull/682 .
            Hide
            tjenness Tim Jenness added a comment -

            Looks great. Thanks for all the cleanups.

            Show
            tjenness Tim Jenness added a comment - Looks great. Thanks for all the cleanups.
            Hide
            salnikov Andy Salnikov added a comment -

            Thanks everyone for reviewing and all suggestions! Finally merged.

            Show
            salnikov Andy Salnikov added a comment - Thanks everyone for reviewing and all suggestions! Finally merged.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Salnikov, Christine Banek, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.