Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34483

Butler Registry to ObsCore exporter



    • Ops Middleware
    • No


      The ObsTAP service requires that we have a database populated with ObsCore entries.

      We need something that can take a Registry (possibly with a dataset query constraint) and exports it to, say, a parquet file that can be ingested into the ObsCore database. Some values can come from configuration (such as the telescope location and how to map a dataset type to a processing level).

      Setting up the ObsCore database and loading these files into it can be done as a separate ticket.

      A proof-of-concept was written at https://github.com/lsst-dm/dax_obscore which attempts to show that this can be done. I don't know if any of this code can be used in the final system but it doesn't seem unreasonable to use the package itself dax_obscore.

      The code in dax_obscore has some problems in that it should never call butler.get – the sky region for an exposure must come from the visit record, as should the exposure time. calexp is visit based so that is fine, but for raw there might be a complication in getting the "footprint" in the absence of a direct link from an exposure record to a visit record.

      I envisage that we have a butler plugin click command that can do the equivalent of query-datasets and dump the results as a parquet table. Obviously we may also want a bespoke exporter that is distinct from the command-line tooling when doing a full export.

      gpdf how are you envisaging we use the visit concept in ObsCore? In theory we don't need it at all and can use the naive approach of every observation is an exposure. How many dataset types is the service meant to expose? raw, calexp and the tract/patch coadds?


        Issue Links


            No work has yet been logged on this issue.


              salnikov Andy Salnikov
              tjenness Tim Jenness
              Tim Jenness
              Andy Salnikov, Christine Banek, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Tim Jenness
              0 Vote for this issue
              6 Start watching this issue




                  No builds found.