Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32694

Split AP pipeline into ApPipeWithFakes

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ap_pipe, ap_verify
    • None

    Description

      Currently, ApPipe.yaml is considered the "base" AP pipeline; it is included by ApVerify.yaml, which is in turn included by ApVerifyWithFakes.yaml. However, this design causes problems for ApVerifyWithFakes, which has a significantly different structure (in particular, a distinction between steps done with and without fakes) and its own rules for which metrics instrument which tasks.

      The problem can be mitigated by splitting ApPipe.yaml into two pipelines: one which is a dependency of ApVerify.yaml, and a new ApPipeWithFakes.yaml that is a dependency of ApVerifyWithFakes.yaml. ApPipe.yaml and ApPipeWithFakes.yaml will not depend on each other, allowing each to have the labels, contracts, and other elements that are appropriate for that pipeline.

      Configurations that should be shared between the two pipelines should be factored out into their own configs, consistent with RFC-775 guidelines.

      Attachments

        Issue Links

          Activity

            krzys Krzysztof Findeisen added a comment - - edited

            DM-32245 added a ApPipeMultiTractFakes.yaml; use this as the basis instead of ApVerifyWithFakes.yaml (there's still some work to do on this ticket, as the current file depends on ApPipe, and ApPipe is not yet multi-tract).

            krzys Krzysztof Findeisen added a comment - - edited DM-32245 added a ApPipeMultiTractFakes.yaml ; use this as the basis instead of ApVerifyWithFakes.yaml (there's still some work to do on this ticket, as the current file depends on ApPipe , and ApPipe is not yet multi-tract).
            krzys Krzysztof Findeisen added a comment - - edited

            See also DM-32988, which formalizes the question of multi-tract processing and will do some pipeline cleanup of its own.

            krzys Krzysztof Findeisen added a comment - - edited See also DM-32988 , which formalizes the question of multi-tract processing and will do some pipeline cleanup of its own.

            sullivan asked that this issue be merged at least one day after DM-34254, as both have changes that could affect completeness metrics.

            krzys Krzysztof Findeisen added a comment - sullivan asked that this issue be merged at least one day after DM-34254 , as both have changes that could affect completeness metrics.

            Thanks for agreeing to look at this, mrawls! The biggest changes ended up being in ap_pipe; I recommend looking at those and ap_verify by individual commits.

            krzys Krzysztof Findeisen added a comment - Thanks for agreeing to look at this, mrawls ! The biggest changes ended up being in ap_pipe ; I recommend looking at those and ap_verify by individual commits.

            Thank you for fixing this up to be much more self-consistent, and to implement a less onerous DM-30210 workaround. Most of the GitHub comments are minor, and provided you can please convince me the camera-specific ApPipeWithFakes pipelines are importing all the right things from all the right places, I am very happy to have you merge this.

            mrawls Meredith Rawls added a comment - Thank you for fixing this up to be much more self-consistent, and to implement a less onerous DM-30210 workaround. Most of the GitHub comments are minor, and provided you can please convince me the camera-specific ApPipeWithFakes pipelines are importing all the right things from all the right places, I am very happy to have you merge this.

            Ok, everything's merged. This should unblock DM-34710 and DM-34623.

            krzys Krzysztof Findeisen added a comment - Ok, everything's merged. This should unblock DM-34710 and DM-34623 .

            People

              krzys Krzysztof Findeisen
              krzys Krzysztof Findeisen
              Meredith Rawls
              Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.