Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-39214

Move ingredient pipeline definitions in ap_pipe and ap_verify to the pipelines directory

    XMLWordPrintable

Details

    • Alert Production
    • No

    Description

      RFC-775 laid out most of a proposal for organizing pipeline YAML files in packages, but left some details the implementation. This included a goal of clearly separating pipelines that should be executed (and for the most part, not imported into other pipelines) from "ingredient" pipelines that are the opposite: they should not be executed, only imported.

      In the implementation of drp_pipe, we walked back one aspect of the RFC-775 proposal, by leaving the "runnable" pipelines as regular YAML files, rather than some sort of expanded/resolved variant thereof, as the latter didn't seem to be a sound concept in end. But we did settle on putting the "don't execute, only import" pipeline YAML files in an "ingredients" directory with (when appropriate) some of the same per-instrument structure as the pipelines directory.

      For consistency, we should make the same change to ap_pipe as well. That may just be a matter of moving pipelines that are not associated with instruments to an "ingredients" directory, but an ap_pipe expert should check that guess.

      Attachments

        Issue Links

          Activity

            Are the pipelines in ap_verify also in scope for this issue, or would they go on a separate one? I expect that pipeline references from ap_verify to ap_pipe would need to be updated anyway.

            krzys Krzysztof Findeisen added a comment - Are the pipelines in ap_verify also in scope for this issue, or would they go on a separate one? I expect that pipeline references from ap_verify to ap_pipe would need to be updated anyway.
            lskelvin Lee Kelvin added a comment -

            My feeling is yes, they should be. The purpose of RFC-927 is to help prevent the two common failure modes from occurring (using ingredients directly, and not using camera-specific YAMLs). It looks like AP_VERIFY would also benefit from this re-organization too, so I would vote to also bring it in scope here.

            If you agree, I can add a note about that on RFC-927.

            lskelvin Lee Kelvin added a comment - My feeling is yes, they should be. The purpose of RFC-927 is to help prevent the two common failure modes from occurring (using ingredients directly, and not using camera-specific YAMLs). It looks like AP_VERIFY would also benefit from this re-organization too, so I would vote to also bring it in scope here. If you agree, I can add a note about that on RFC-927 .
            lskelvin Lee Kelvin added a comment -

            Local unit test runs for ap_pipe, ap_verify and drp_pipe (for test_pipelines) all pass successfully.

            AP_PIPE PR: https://github.com/lsst/ap_pipe/pull/137

            AP_VERIFY PR: https://github.com/lsst/ap_verify/pull/186

            Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38885/pipeline

            lskelvin Lee Kelvin added a comment - Local unit test runs for ap_pipe , ap_verify and drp_pipe (for test_pipelines ) all pass successfully. AP_PIPE PR: https://github.com/lsst/ap_pipe/pull/137 AP_VERIFY PR: https://github.com/lsst/ap_verify/pull/186 Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38885/pipeline
            lskelvin Lee Kelvin added a comment -

            Thanks krzys for your comments on the PRs, much appreciated. Jenkins is happy, branches merged and deleted .

            lskelvin Lee Kelvin added a comment - Thanks krzys for your comments on the PRs, much appreciated. Jenkins is happy, branches merged and deleted .

            People

              lskelvin Lee Kelvin
              jbosch Jim Bosch
              Krzysztof Findeisen
              Jim Bosch, Krzysztof Findeisen, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.