Fix Version/s: None
Component/s: daf_butler, dax_obscore
The ObsTAP service requires that we have a database populated with ObsCore entries.
We need something that can take a Registry (possibly with a dataset query constraint) and exports it to, say, a parquet file that can be ingested into the ObsCore database. Some values can come from configuration (such as the telescope location and how to map a dataset type to a processing level).
Setting up the ObsCore database and loading these files into it can be done as a separate ticket.
A proof-of-concept was written at https://github.com/lsst-dm/dax_obscore which attempts to show that this can be done. I don't know if any of this code can be used in the final system but it doesn't seem unreasonable to use the package itself dax_obscore.
The code in dax_obscore has some problems in that it should never call butler.get – the sky region for an exposure must come from the visit record, as should the exposure time. calexp is visit based so that is fine, but for raw there might be a complication in getting the "footprint" in the absence of a direct link from an exposure record to a visit record.
I envisage that we have a butler plugin click command that can do the equivalent of query-datasets and dump the results as a parquet table. Obviously we may also want a bespoke exporter that is distinct from the command-line tooling when doing a full export.
Gregory Dubois-Felsmann how are you envisaging we use the visit concept in ObsCore? In theory we don't need it at all and can use the naive approach of every observation is an exposure. How many dataset types is the service meant to expose? raw, calexp and the tract/patch coadds?
- is blocked by
DM-22496 Provide an interim definition of the obs_publisher_did field in ObsCore that is usable for Gen3-based image data
- In Progress
- is triggering
DM-34735 Add UUID support to URL format string
DM-34685 Create production Butler-to-ObsCore configuration file(s) in sdm_schemas/obscore
- Wiki Page
Wiki Page Loading...
Gregory Dubois-Felsmann, I have made a fresh copy of CSV from /repo/main using example.yaml configuration, it is at https://lsst.ncsa.illinois.edu/~salnikov/data.csv.gz, or in /project/salnikov/obscore/data.csv.gz on lsst-devl.
Tim Jenness, sorry, I forgot to make PR for daf_butler earlier, it needed a small fix to avoid crashing: https://github.com/lsst/daf_butler/pull/682.
Andy Salnikov I have (finally) been looking closely at this, looking at what was released a while back (on 4/24).
Do you have a CSV version of the most recent export format?