Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27099

Re-bridge ap_verify and ctrl_mpexec

    XMLWordPrintable

    Details

      Description

      In Gen 3 mode, ap_verify relies on ctrl_mpexec for running the AP pipeline, but there is as yet no API to support pipeline execution from within a larger program (DM-26239). ap_verify hacked around this by calling CmdLineFwk's command-line parser, but this code was removed in DM-26874.

      Following discussion on #dm-middleware, attempt to use click.testing.CliRunner as an almost-drop-in replacement for CmdLineFwk.parseAndRun. If this breaks important pipeline features (parallelism, logging, etc.), fall back to manually calling the scripts at https://github.com/lsst/ctrl_mpexec/tree/master/python/lsst/ctrl/mpexec/cli/script.

        Attachments

          Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. DM-27117 Add dummy pipeline to ap_verify testing

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-31180 ap_verify loses logs from pipeline

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26874 rename pipetask2 to pipetask, remove old pipetask

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-26239 Create Python API for pipetask

          • To Do

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Epic Link DM-26803 [ 439755 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-26239 [ DM-26239 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-26874 [ DM-26874 ]
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            If time permits, consider changing test_driver.py to mock only at lower levels; previously, all of CmdLineFwk was replaced with a mock, which is why the problem didn't get caught in unit tests.

            Show
            krzys Krzysztof Findeisen added a comment - If time permits, consider changing test_driver.py to mock only at lower levels; previously, all of CmdLineFwk was replaced with a mock, which is why the problem didn't get caught in unit tests.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            CliRunner seems to work fine; all expected logs appear on the terminal and there aren't any (obvious) concurrency problems.

            Show
            krzys Krzysztof Findeisen added a comment - CliRunner seems to work fine; all expected logs appear on the terminal and there aren't any (obvious) concurrency problems.
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-27117 [ DM-27117 ]
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Nate Pease [X], would you be willing to review this?

            I ended up deferring the testing question to DM-27117, since it would involve a significant rewrite of the existing tests. However, this means for that now the main Gen 3 thing that is still covered in ap_verify's unit tests is importability.

            Show
            krzys Krzysztof Findeisen added a comment - Nate Pease [X] , would you be willing to review this? I ended up deferring the testing question to DM-27117 , since it would involve a significant rewrite of the existing tests. However, this means for that now the main Gen 3 thing that is still covered in ap_verify 's unit tests is importability.
            krzys Krzysztof Findeisen made changes -
            Reviewers Nate Pease [ npease ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            npease Nate Pease [X] (Inactive) added a comment -

            looks good.

            Show
            npease Nate Pease [X] (Inactive) added a comment - looks good.
            npease Nate Pease [X] (Inactive) made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-31180 [ DM-31180 ]

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Nate Pease [X] (Inactive)
              Watchers:
              Ian Sullivan, Krzysztof Findeisen, Nate Pease [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.