Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29543

Gen3 refcat converter

    XMLWordPrintable

    Details

      Description

      Currently, the only way to get a gen3 refcat is to ingest it into a gen2 repo an then convert it to gen3. IngestIndexedReferenceTask is a CmdLineTask, and we need a PipelineTask refcat ingester. Much of the internals are the same, so we can probably do it with a shared base class, but gen3 has additional information loaded into the registry about the shards, etc. that will take some care. The multiprocessing manager code, IngestIndexManager, may not be compatible with the gen3 multiprocessing system: it's designed to do all of its own multiprocessing internally, without an external controller.

      These docs about how to ingest refcat data will probably be useful in putting together the gen3 code: https://pipelines.lsst.io/modules/lsst.meas.algorithms/creating-a-reference-catalog.html

      I also wrote a custom Gaia ingester (IngestGaiaManager and IngestGaiaReferenceTask) to handle the gaia fluxes, which might be able to work "for free" with the new system.

        Attachments

          Issue Links

            Activity

            No work has yet been logged on this issue.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Eli Rykoff
              Watchers:
              Dan Taranu, Eli Rykoff, Gregory Dubois-Felsmann, Ian Sullivan, James Chiang, Jim Bosch, John Parejko, Krzysztof Findeisen, Lee Kelvin, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.