Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21919

Run ap_verify end-to-end in Gen 3

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP S20-6 (May), AP F20-1 (June), AP F20-2 (July), AP F20-3 (August)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Once all the individual components support Gen 3, add a command-line option to ap_verify.py that ingests into a Gen 3 repository, then runs the AP pipeline and any metric tasks as Gen 3 products. Design the UI so that Gen 3 can naturally become the default behavior as Gen 2 is phased out.

        Attachments

          Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. DM-26237 Can't rerun ap_verify on same repository in Gen 3

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21888 Stand up Gen3 version of ap_verify HiTS (2015) CI test

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24262 Run HSC AP processing in CI using Gen 3

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26028 Generalize ap_verify to use other executors

          • Won't Fix

          Improvement - An improvement or enhancement to an existing feature or task. DM-26140 Centralize Gen 3 pipeline configuration info for ap_verify datasets

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21885 Convert concrete MetricTasks to PipelineTasks

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21915 Support Gen 3 ingestion of ap_verify datasets

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21939 Create Gen 3 AP Pipeline

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-26070 Add visit definition to ap_verify

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Link This issue blocks DM-21888 [ DM-21888 ]
            krzys Krzysztof Findeisen made changes -
            Summary Give ap_verify.py a "Gen 3" mode Give ap_verify.py a "Gen 3 mode"
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-21886 [ DM-21886 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-21885 [ DM-21885 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-21915 [ DM-21915 ]
            krzys Krzysztof Findeisen made changes -
            Rank Ranked higher
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-21939 [ DM-21939 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-21886 [ DM-21886 ]
            krzys Krzysztof Findeisen made changes -
            Summary Give ap_verify.py a "Gen 3 mode" Run ap_verify end-to-end in Gen 3
            krzys Krzysztof Findeisen made changes -
            Epic Link DM-22633 [ 427742 ]
            swinbank John Swinbank made changes -
            Sprint AP S20-6 (May) [ 987 ]
            Urgent? off
            swinbank John Swinbank made changes -
            Link This issue blocks DM-24262 [ DM-24262 ]
            swinbank John Swinbank made changes -
            Epic Link DM-22633 [ 427742 ] DM-24341 [ 433028 ]
            swinbank John Swinbank made changes -
            Epic Link DM-24341 [ 433028 ] DM-25145 [ 435263 ]
            swinbank John Swinbank made changes -
            Sprint AP S20-6 (May) [ 987 ] AP S20-6 (May), AP F20-1 (June) [ 987, 1019 ]
            krzys Krzysztof Findeisen made changes -
            Story Points 2 6
            swinbank John Swinbank made changes -
            Sprint AP S20-6 (May), AP F20-1 (June) [ 987, 1019 ] AP S20-6 (May), AP F20-1 (June), AP F20-2 (July) [ 987, 1019, 1025 ]
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-26028 [ DM-26028 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-26070 [ DM-26070 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-26140 [ DM-26140 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-26237 [ DM-26237 ]
            swinbank John Swinbank made changes -
            Sprint AP S20-6 (May), AP F20-1 (June), AP F20-2 (July) [ 987, 1019, 1025 ] AP S20-6 (May), AP F20-1 (June), AP F20-2 (July), AP F20-3 (August) [ 987, 1019, 1025, 1033 ]
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            This is another ticket that sprawled out of control. I'd like to divide up the review three ways:

            Show
            krzys Krzysztof Findeisen added a comment - - edited This is another ticket that sprawled out of control. I'd like to divide up the review three ways: Nate Lust , could you review the changes to ctrl_mpexec , pipe_base , verify , ip_diffim , ap_association , and ap_pipe ? It's a total of 75 lines. Simon Krughoff , could you review the first three ap_verify commits ("Make Workspace...", "Create verification pipelines", "Rename runApPipe..."), for a total of 200 lines? Christopher Waters , could you review the last three ap_verify commits ("Add Gen 3 pipeline...", "Implement --gen3...", "Add pipeline override..."), for a total of 300 lines?
            krzys Krzysztof Findeisen made changes -
            Reviewers Christopher Waters, Nate Lust, Simon Krughoff [ cwaters, nlust, krughoff ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            czw Christopher Waters added a comment -

            I've made a couple of minor comments on my changeset on github.  I'll remove myself from the reviewer list, but feel free to re-add me if other changes make what I reviewed out-of-date.

            Show
            czw Christopher Waters added a comment - I've made a couple of minor comments on my changeset on github.  I'll remove myself from the reviewer list, but feel free to re-add me if other changes make what I reviewed out-of-date.
            czw Christopher Waters made changes -
            Reviewers Christopher Waters, Nate Lust, Simon Krughoff [ cwaters, nlust, krughoff ] Nate Lust, Simon Krughoff [ nlust, krughoff ]
            Hide
            Permalink
            nlust Nate Lust added a comment -

            I think I marked all my stuff reviewed and will take myself off as a reviewer. If I missed a pull request let me know and I will look at it. All looks good with one minor comment.

            Show
            nlust Nate Lust added a comment - I think I marked all my stuff reviewed and will take myself off as a reviewer. If I missed a pull request let me know and I will look at it. All looks good with one minor comment.
            nlust Nate Lust made changes -
            Reviewers Nate Lust, Simon Krughoff [ nlust, krughoff ] Simon Krughoff [ krughoff ]
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            Krzysztof Findeisen sorry for the delay. I have just a couple of comments. They are essentially clarifying comments. In general it looks good to me.

            Show
            krughoff Simon Krughoff added a comment - Krzysztof Findeisen sorry for the delay. I have just a couple of comments. They are essentially clarifying comments. In general it looks good to me.
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            Since I'm the last reviewer, I'm marking as complete. Let me know if that's not what I should do.

            Show
            krughoff Simon Krughoff added a comment - Since I'm the last reviewer, I'm marking as complete. Let me know if that's not what I should do.
            krughoff Simon Krughoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Simon Krughoff
              Watchers:
              Christopher Waters, Krzysztof Findeisen, Nate Lust, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.