Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13163

Refactor ap_pipe to use CmdLineTask primitives

    XMLWordPrintable

    Details

    • Story Points:
      10
    • Epic Link:
    • Sprint:
      AP S18-2, AP S18-3, AP S18-4
    • Team:
      Alert Production

      Description

      Because ap_pipe was developed with a CI verification workflow in mind (in concert with ap_verify), it includes ingestion and does not use the Stack primitives. We are moving away from this approach; this ticket is refactor ap_pipe to have a separate ingestion routine as needed and to have the remaining processing be a CmdLineTask.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13164 Update documentation for ap_pipe

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12549 ap_pipe must call AssociationTask in a reproducible order

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-13492 Remove --rerun argument for ap_verify

          • Done
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. DM-12557 False positives when testing for pipeline completion

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12314 Generalize ap_pipe to multiple visits

          • Invalid
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13500 Ingest ap_verify data into single repository

          • Won't Fix
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13451 Make ap_verify responsible for ingestion

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13535 Accept idiomatic input repositories

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13536 Use repositories more idiomatically

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13567 Add tests to ap_pipe

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-13626 Butler inconsistently resolves duplicate repositories

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11767 Write the formatter for serializing DIAObjects/Sources through the butler

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11865 ImageDifference should allow coadd templates and output to live in different repos

          • Won't Fix

          Improvement - An improvement or enhancement to an existing feature or task. DM-13602 Set association DB location at run time, not config time

          • To Do

          Improvement - An improvement or enhancement to an existing feature or task. DM-13672 Make AssociationTask stateless

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            Progress report: ap_pipe is now a CmdLineTask, although it requires quite a few hacks (including significant duplication of pipe_base code) to deal with DM-11767, DM-11865, DM-13602, DM-13626, and DM-13672. We'll see how horrified the reviewer is.

            I have not yet made any attempt to modify ap_verify to work with the new ap_pipe API. Since the new ap_pipe drivers take datarefs as inputs, this will require adding some butler code to ap_verify.

            The command line for ap_pipe has changed slightly; see https://gist.github.com/kfindeisen/2d3d4fcc7252b727761f172a59d15c48#file-gistfile1-txt for examples.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Progress report: ap_pipe is now a CmdLineTask , although it requires quite a few hacks (including significant duplication of pipe_base code) to deal with DM-11767 , DM-11865 , DM-13602 , DM-13626 , and DM-13672 . We'll see how horrified the reviewer is. I have not yet made any attempt to modify ap_verify to work with the new ap_pipe API. Since the new ap_pipe drivers take datarefs as inputs, this will require adding some butler code to ap_verify . The command line for ap_pipe has changed slightly; see https://gist.github.com/kfindeisen/2d3d4fcc7252b727761f172a59d15c48#file-gistfile1-txt for examples.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Hi Russell Owen, can you please review this refactoring of ap_pipe? I'd be particularly interested in any misuse of the task framework that wasn't deliberate (i.e., flagged by a comment).

            Show
            krzys Krzysztof Findeisen added a comment - Hi Russell Owen , can you please review this refactoring of ap_pipe ? I'd be particularly interested in any misuse of the task framework that wasn't deliberate (i.e., flagged by a comment).
            Hide
            Permalink
            rowen Russell Owen added a comment -

            My main request is to fix DM-11865 by adding support to pipe_base ArgumentParser for additional repo arguments. I think it would be fairly simple to do that and it would allow you to remove a lot of duplicate code and "TODO" work, leaving the new code in ap_pipe much cleaner and easier to follow.

            I also suggest that you make ingest a task (a subclass of pipe_base Task) as it is easy to do and would simplify some of the things you are doing now and hopefully eliminate the need to pass fullMetadata around.

            Other than that it looks good. Thank you for the excellent documentation.

            Show
            rowen Russell Owen added a comment - My main request is to fix DM-11865 by adding support to pipe_base ArgumentParser for additional repo arguments. I think it would be fairly simple to do that and it would allow you to remove a lot of duplicate code and "TODO" work, leaving the new code in ap_pipe much cleaner and easier to follow. I also suggest that you make ingest a task (a subclass of pipe_base Task) as it is easy to do and would simplify some of the things you are doing now and hopefully eliminate the need to pass fullMetadata around. Other than that it looks good. Thank you for the excellent documentation.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            Thank you for the helpful review. Conversion of the ap_verify's ingestion module into a task will happen on DM-13530, as it would be a significant scope change for this ticket (in particular, reimplementing ingestion does not affect ap_pipe in any way). Adding multi-repository support to pipe_base is, as agreed on the ap_pipe pull request, something we will RFC separately. I think I've addressed everything else.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Thank you for the helpful review. Conversion of the ap_verify 's ingestion module into a task will happen on DM-13530 , as it would be a significant scope change for this ticket (in particular, reimplementing ingestion does not affect ap_pipe in any way). Adding multi-repository support to pipe_base is, as agreed on the ap_pipe pull request, something we will RFC separately. I think I've addressed everything else.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              ebellm Eric Bellm
              Reviewers:
              Russell Owen
              Watchers:
              Eric Bellm, Krzysztof Findeisen, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.