Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-554

Refactoring of pipe_supertask

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      PipelineTask execution activator/framework needs serious refactoring to make it more reusable and to support multiple execution environments. Currently we have all code in  pipe_supertask package, which is not released yet, and it only supports "laptop activator" application (multiprocess-based activator à la CmdLineTask), which also incorporates support for building/serializing pipelines and QuantumGraphs.

      To make it more reusable I propose to split current code into two packages:

      • Common code usable by all clients will be moved to pipe_base (modules include pipeline, graph, pipeTools, pipelineBuilder, graphBuilder, expr_parser), some code will need small scale migration to make it independent of argparse.Namespace
      • Remaining code that implements MP activator will be moved to a new package named ctrl_mpexec, pieces of that code will be reusable by other activators as well. This new package will be included into lsst_distrib. [2]
      • MP activator application (in ctrl_mpexec), which can also execute generic pipeline/graph tools, will be renamed from stac to pipe_tools (suggestions for better name are welcome).

      I do not expect this to be the last round of refactoring/renaming, we may get better idea of how do we have to structure the code as we continue implementing new activators, this refactoring is just a starting point to unblock that future development.


      [1] Slack discussion link: https://lsstc.slack.com/archives/CAVQQ6SBX/p1542671318061700)

      [2] Potentially reusable code may be better to package in a separate package, but the amount of code is small and non-reusable code should not have extra dependencies, so I prefer to keep all of that in a single package for now.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            All sounds reasonable.  While I'm hoping someone else will chime in with an even better name, I think I'd prefer pipe_tool to pipe_tools.

            Also, I'm going to go ahead and bring that naming discussion into #dm-naming-things on slack, as that might give it more visibility and hence suggestions.

            Show
            jbosch Jim Bosch added a comment - All sounds reasonable.  While I'm hoping someone else will chime in with an even better name, I think I'd prefer pipe_tool to pipe_tools . Also, I'm going to go ahead and bring that naming discussion into #dm-naming-things on slack, as that might give it more visibility and hence suggestions.
            Hide
            salnikov Andy Salnikov added a comment -

            In the #dm-naming-things discussion pipetask was suggested as a better name for stac application (with pipetool probably being a next contender, and despair maybe in a tie with it ).

            I'm going to adopt the RFC with this proposed name.

            Show
            salnikov Andy Salnikov added a comment - In the #dm-naming-things discussion pipetask was suggested as a better name for stac application (with pipetool probably being a next contender, and despair maybe in a tie with it  ). I'm going to adopt the RFC with this proposed name.
            Hide
            salnikov Andy Salnikov added a comment -

            Adopting as proposed, except for stac replacement is going to be called pipetask.

            Show
            salnikov Andy Salnikov added a comment - Adopting as proposed, except for stac replacement is going to be called pipetask .

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              salnikov Andy Salnikov
              Watchers:
              Andy Salnikov, Jim Bosch, John Swinbank, Michelle Gower
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.