Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-39214

Move ingredient pipeline definitions in ap_pipe and ap_verify to the pipelines directory

    XMLWordPrintable

    Details

    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      RFC-775 laid out most of a proposal for organizing pipeline YAML files in packages, but left some details the implementation. This included a goal of clearly separating pipelines that should be executed (and for the most part, not imported into other pipelines) from "ingredient" pipelines that are the opposite: they should not be executed, only imported.

      In the implementation of drp_pipe, we walked back one aspect of the RFC-775 proposal, by leaving the "runnable" pipelines as regular YAML files, rather than some sort of expanded/resolved variant thereof, as the latter didn't seem to be a sound concept in end. But we did settle on putting the "don't execute, only import" pipeline YAML files in an "ingredients" directory with (when appropriate) some of the same per-instrument structure as the pipelines directory.

      For consistency, we should make the same change to ap_pipe as well. That may just be a matter of moving pipelines that are not associated with instruments to an "ingredients" directory, but an ap_pipe expert should check that guess.

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue relates to RFC-775 [ RFC-775 ]
            jbosch Jim Bosch made changes -
            Link This issue is triggered by RFC-927 [ RFC-927 ]
            Hide
            krzys Krzysztof Findeisen added a comment -

            Are the pipelines in ap_verify also in scope for this issue, or would they go on a separate one? I expect that pipeline references from ap_verify to ap_pipe would need to be updated anyway.

            Show
            krzys Krzysztof Findeisen added a comment - Are the pipelines in ap_verify also in scope for this issue, or would they go on a separate one? I expect that pipeline references from ap_verify to ap_pipe would need to be updated anyway.
            Hide
            lskelvin Lee Kelvin added a comment -

            My feeling is yes, they should be. The purpose of RFC-927 is to help prevent the two common failure modes from occurring (using ingredients directly, and not using camera-specific YAMLs). It looks like AP_VERIFY would also benefit from this re-organization too, so I would vote to also bring it in scope here.

            If you agree, I can add a note about that on RFC-927.

            Show
            lskelvin Lee Kelvin added a comment - My feeling is yes, they should be. The purpose of RFC-927 is to help prevent the two common failure modes from occurring (using ingredients directly, and not using camera-specific YAMLs). It looks like AP_VERIFY would also benefit from this re-organization too, so I would vote to also bring it in scope here. If you agree, I can add a note about that on RFC-927 .
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-39333 [ DM-39333 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-36796 [ DM-36796 ]
            lskelvin Lee Kelvin made changes -
            Summary Move no-instrument pipelines in ap_pipe to an "ingredients" directory Move ingredient pipeline definitions in ap_pipe to the pipelines directory
            lskelvin Lee Kelvin made changes -
            Component/s ap_verify [ 14167 ]
            lskelvin Lee Kelvin made changes -
            Assignee Lee Kelvin [ lskelvin ]
            lskelvin Lee Kelvin made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            lskelvin Lee Kelvin added a comment -

            Local unit test runs for ap_pipe, ap_verify and drp_pipe (for test_pipelines) all pass successfully.

            AP_PIPE PR: https://github.com/lsst/ap_pipe/pull/137

            AP_VERIFY PR: https://github.com/lsst/ap_verify/pull/186

            Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38885/pipeline

            Show
            lskelvin Lee Kelvin added a comment - Local unit test runs for ap_pipe , ap_verify and drp_pipe (for test_pipelines ) all pass successfully. AP_PIPE PR: https://github.com/lsst/ap_pipe/pull/137 AP_VERIFY PR: https://github.com/lsst/ap_verify/pull/186 Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38885/pipeline
            lskelvin Lee Kelvin made changes -
            Reviewers Krzysztof Findeisen [ krzys ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            lskelvin Lee Kelvin made changes -
            Summary Move ingredient pipeline definitions in ap_pipe to the pipelines directory Move ingredient pipeline definitions in ap_pipe and ap_verify into the pipelines directory
            lskelvin Lee Kelvin made changes -
            Summary Move ingredient pipeline definitions in ap_pipe and ap_verify into the pipelines directory Move ingredient pipeline definitions in ap_pipe and ap_verify to the pipelines directory
            krzys Krzysztof Findeisen made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks Krzysztof Findeisen for your comments on the PRs, much appreciated. Jenkins is happy, branches merged and deleted .

            Show
            lskelvin Lee Kelvin added a comment - Thanks Krzysztof Findeisen for your comments on the PRs, much appreciated. Jenkins is happy, branches merged and deleted .
            lskelvin Lee Kelvin made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-39465 [ DM-39465 ]

              People

              Assignee:
              lskelvin Lee Kelvin
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Krzysztof Findeisen
              Watchers:
              Jim Bosch, Krzysztof Findeisen, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.