Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34483

Butler Registry to ObsCore exporter



    • Team:
      Ops Middleware
    • Urgent?:


      The ObsTAP service requires that we have a database populated with ObsCore entries.

      We need something that can take a Registry (possibly with a dataset query constraint) and exports it to, say, a parquet file that can be ingested into the ObsCore database. Some values can come from configuration (such as the telescope location and how to map a dataset type to a processing level).

      Setting up the ObsCore database and loading these files into it can be done as a separate ticket.

      A proof-of-concept was written at https://github.com/lsst-dm/dax_obscore which attempts to show that this can be done. I don't know if any of this code can be used in the final system but it doesn't seem unreasonable to use the package itself dax_obscore.

      The code in dax_obscore has some problems in that it should never call butler.get – the sky region for an exposure must come from the visit record, as should the exposure time. calexp is visit based so that is fine, but for raw there might be a complication in getting the "footprint" in the absence of a direct link from an exposure record to a visit record.

      I envisage that we have a butler plugin click command that can do the equivalent of query-datasets and dump the results as a parquet table. Obviously we may also want a bespoke exporter that is distinct from the command-line tooling when doing a full export.

      Gregory Dubois-Felsmann how are you envisaging we use the visit concept in ObsCore? In theory we don't need it at all and can use the naive approach of every observation is an exposure. How many dataset types is the service meant to expose? raw, calexp and the tract/patch coadds?


          Issue Links


            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Remote Link This issue links to "Wiki Page (Confluence)" [ 32730 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-22496 [ DM-22496 ]
            salnikov Andy Salnikov made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Team Data Access and Database [ 10204 ] Ops Middleware [ 15600 ]
            salnikov Andy Salnikov made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            gpdf Gregory Dubois-Felsmann made changes -
            Link This issue is triggering DM-34685 [ DM-34685 ]
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            salnikov Andy Salnikov made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            salnikov Andy Salnikov made changes -
            Link This issue is triggering DM-34735 [ DM-34735 ]
            salnikov Andy Salnikov made changes -
            Component/s dax_obscore [ 20400 ]
            salnikov Andy Salnikov made changes -
            Labels gen3-middleware ObsCore gen3-middleware


              salnikov Andy Salnikov
              tjenness Tim Jenness
              Tim Jenness
              Andy Salnikov, Christine Banek, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Tim Jenness
              0 Vote for this issue
              6 Start watching this issue




                  No builds found.