Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29543

Gen3 refcat converter

    XMLWordPrintable

    Details

      Description

      Currently, the only way to get a gen3 refcat is to ingest it into a gen2 repo an then convert it to gen3. IngestIndexedReferenceTask is a CmdLineTask, and we need a PipelineTask refcat ingester. Much of the internals are the same, so we can probably do it with a shared base class, but gen3 has additional information loaded into the registry about the shards, etc. that will take some care. The multiprocessing manager code, IngestIndexManager, may not be compatible with the gen3 multiprocessing system: it's designed to do all of its own multiprocessing internally, without an external controller.

      These docs about how to ingest refcat data will probably be useful in putting together the gen3 code: https://pipelines.lsst.io/modules/lsst.meas.algorithms/creating-a-reference-catalog.html

      I also wrote a custom Gaia ingester (IngestGaiaManager and IngestGaiaReferenceTask) to handle the gaia fluxes, which might be able to work "for free" with the new system.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-20572 Remove "Generation 2" Middleware

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-32389 Create refcat management script for ap_verify datasets

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31320 Prep work for creating Gen 3 refcat ingester

          • Done
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31698 Move ConvertReferenceCatalog classes out of ingestIndex file

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31817 Remove default value on DatasetConfig.ref_dataset_name

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31704 Design and document reference catalog dataset policy

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30624 Document gen2 to gen3 refcat ingestion

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33639 Fix refcat ingest docs and convert message to ingest into a RUN collection

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31367 Add butler register-dataset-type subcommand

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Link This issue blocks DM-20572 [ DM-20572 ]
            krzys Krzysztof Findeisen made changes -
            Labels SciencePipelines SciencePipelines gen2-deprecation-debt
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-29857 [ DM-29857 ]
            sullivan Ian Sullivan made changes -
            Assignee John Parejko [ parejkoj ]
            sullivan Ian Sullivan made changes -
            Sprint AP S21-7 (June) [ 1096 ]
            Team Alert Production [ 10300 ]
            sullivan Ian Sullivan made changes -
            Epic Link DM-29209 [ 459201 ]
            sullivan Ian Sullivan made changes -
            Labels SciencePipelines gen2-deprecation-debt SciencePipelines ap-analysis gen2-deprecation-debt
            sullivan Ian Sullivan made changes -
            Sprint AP F21-1 (June) [ 1096 ] AP F21-2 (July) [ 1102 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked higher
            Parejkoj John Parejko made changes -
            Description Currently, the only way to get a gen3 refcat is to ingest it into a gen2 repo an then convert it to gen3. {{IngestIndexedReferenceTask}} is a CmdLineTask, and we need a PipelineTask refcat ingester. Much of the internals are the same, so we can probably do it with a shared base class, but gen3 has additional information loaded into the registry about the shards, etc. that will take some care. The multiprocessing manager code, {{IngestIndexManager}}, may not be compatible with the gen3 multiprocessing system: it's designed to do all of its own multiprocessing internally, without an external controller.

            There docs about how to ingest refcat data will probably be useful in putting together the gen3 code: https://pipelines.lsst.io/modules/lsst.meas.algorithms/creating-a-reference-catalog.html

            I also wrote a custom Gaia ingester (IngestGaiaManager and IngestGaiaReferenceTask) to handle the gaia fluxes, which might be able to work "for free" with the new system.             		Currently, the only way to get a gen3 refcat is to ingest it into a gen2 repo an then convert it to gen3. {{IngestIndexedReferenceTask}} is a CmdLineTask, and we need a PipelineTask refcat ingester. Much of the internals are the same, so we can probably do it with a shared base class, but gen3 has additional information loaded into the registry about the shards, etc. that will take some care. The multiprocessing manager code, {{IngestIndexManager}}, may not be compatible with the gen3 multiprocessing system: it's designed to do all of its own multiprocessing internally, without an external controller.

            These docs about how to ingest refcat data will probably be useful in putting together the gen3 code: https://pipelines.lsst.io/modules/lsst.meas.algorithms/creating-a-reference-catalog.html

            I also wrote a custom Gaia ingester (IngestGaiaManager and IngestGaiaReferenceTask) to handle the gaia fluxes, which might be able to work "for free" with the new system.
            sullivan Ian Sullivan made changes -
            Epic Link DM-29209 [ 459201 ] DM-30431 [ 504819 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-30624 [ DM-30624 ]
            sullivan Ian Sullivan made changes -
            Story Points 6
            tjenness Tim Jenness made changes -
            Labels SciencePipelines ap-analysis gen2-deprecation-debt SciencePipelines ap-analysis gen2-deprecation-debt gen3-middleware
            Parejkoj John Parejko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            sullivan Ian Sullivan made changes -
            Link This issue is blocked by DM-31320 [ DM-31320 ]
            sullivan Ian Sullivan made changes -
            Sprint AP F21-2 (July) [ 1102 ] AP F21-3 (August) [ 1109 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-31367 [ DM-31367 ]
            sullivan Ian Sullivan made changes -
            Sprint AP F21-3 (August) [ 1109 ] AP F21-4 (September) [ 1113 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked lower
            Parejkoj John Parejko made changes -
            Watchers Gregory Dubois-Felsmann, Ian Sullivan, James Chiang, Jim Bosch, John Parejko, Krzysztof Findeisen, Lee Kelvin, Meredith Rawls, Tim Jenness [ Gregory Dubois-Felsmann, Ian Sullivan, James Chiang, Jim Bosch, John Parejko, Krzysztof Findeisen, Lee Kelvin, Meredith Rawls, Tim Jenness ] Dan Taranu, Gregory Dubois-Felsmann, Ian Sullivan, James Chiang, Jim Bosch, John Parejko, Krzysztof Findeisen, Lee Kelvin, Meredith Rawls, Tim Jenness [ Dan Taranu, Gregory Dubois-Felsmann, Ian Sullivan, James Chiang, Jim Bosch, John Parejko, Krzysztof Findeisen, Lee Kelvin, Meredith Rawls, Tim Jenness ]
            sullivan Ian Sullivan made changes -
            Epic Link DM-30431 [ 504819 ] DM-30514 [ 510188 ]
            Parejkoj John Parejko made changes -
            Link This issue blocks DM-31698 [ DM-31698 ]
            Parejkoj John Parejko made changes -
            Link This issue blocks DM-31698 [ DM-31698 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-31698 [ DM-31698 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-31704 [ DM-31704 ]
            Parejkoj John Parejko made changes -
            Summary Gen3 refcat ingester Gen3 refcat converter
            Parejkoj John Parejko made changes -
            Reviewers Eli Rykoff [ erykoff ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-31817 [ DM-31817 ]
            erykoff Eli Rykoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked higher
            sullivan Ian Sullivan made changes -
            Story Points 6 10
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-32389 [ DM-32389 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-29857 [ DM-29857 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-33639 [ DM-33639 ]

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Eli Rykoff
              Watchers:
              Dan Taranu, Eli Rykoff, Gregory Dubois-Felsmann, Ian Sullivan, James Chiang, Jim Bosch, John Parejko, Krzysztof Findeisen, Lee Kelvin, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.