Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21915

Support Gen 3 ingestion of ap_verify datasets

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP S20-5 (April), AP S20-6 (May), AP F20-1 (June), AP F20-2 (July)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      lsst.ap.verify.DatasetIngestTask is responsible for ingesting an ap_verify dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in ingest_dataset.py that lets users choose between the two.

      Note that the current dataset format includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

      Some overlap with DM-21862, which wants to use HiTS to run single-frame processing.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21919 Run ap_verify end-to-end in Gen 3

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-25806 Support parallel ap_verify ingestion in Gen 3

          • Done
          has to be done after

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24260 Create Gen3 versions of ap_verify datasets

          • Done
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. DM-25786 Cannot import sphgeom objects in Gen 3

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24844 Migrate ap_verify_testdata to obs_lsst

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-21016 Handle DECam instrument signature data in gen3

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-25945 Registry.queryCollections raises MissingCollectionError

          • To Do

          Bug - A problem which impairs or prevents the functions of the product. DM-25970 ap_verify CI command line broken

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21862 Extend Gen3 butler support for obs_decam

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23616 Run converted ap_verify testdata through gen3 pipeline

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21898 Create command-line tools for Gen3 repo administration

          • Won't Fix

          Improvement - An improvement or enhancement to an existing feature or task. DM-24314 Support following links in Butler operations

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-24365 Add relative symbolic link transfer mode to Gen 3 ingest

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21939 Create Gen 3 AP Pipeline

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-26070 Add visit definition to ap_verify

          • Done

            Activity

            No work has yet been logged on this issue.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, John Parejko, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.