Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
PipelineTask execution activator/framework needs serious refactoring to make it more reusable and to support multiple execution environments. Currently we have all code in pipe_supertask package, which is not released yet, and it only supports "laptop activator" application (multiprocess-based activator à la CmdLineTask), which also incorporates support for building/serializing pipelines and QuantumGraphs.
To make it more reusable I propose to split current code into two packages:
- Common code usable by all clients will be moved to pipe_base (modules include pipeline, graph, pipeTools, pipelineBuilder, graphBuilder, expr_parser), some code will need small scale migration to make it independent of argparse.Namespace
- Remaining code that implements MP activator will be moved to a new package named ctrl_mpexec, pieces of that code will be reusable by other activators as well. This new package will be included into lsst_distrib. [2]
- MP activator application (in ctrl_mpexec), which can also execute generic pipeline/graph tools, will be renamed from stac to pipe_tools (suggestions for better name are welcome).
I do not expect this to be the last round of refactoring/renaming, we may get better idea of how do we have to structure the code as we continue implementing new activators, this refactoring is just a starting point to unblock that future development.
[1] Slack discussion link: https://lsstc.slack.com/archives/CAVQQ6SBX/p1542671318061700)
[2] Potentially reusable code may be better to package in a separate package, but the amount of code is small and non-reusable code should not have extra dependencies, so I prefer to keep all of that in a single package for now.
All sounds reasonable. While I'm hoping someone else will chime in with an even better name, I think I'd prefer pipe_tool to pipe_tools.
Also, I'm going to go ahead and bring that naming discussion into #dm-naming-things on slack, as that might give it more visibility and hence suggestions.