Pipelines are defined in YAML files, rather than Python code, so we can't use our usual Sphinx-based tooling to generate documentation for them. This ticket will provide tooling (in pipe_base) and a real-world demonstration of its use (in drp_pipe) that builds rST docs from those YAML files for Sphinx to compile.
In this first version, I'll use SCons to invoke these tools; that should ensure that the generated rST files are installed with the rest of the package when documenteer's stack-docs finds it later. It may be cleaner to call the same tooling directly from a Sphinx plugin in the future, but I'll leave that for another ticket (aside from trying to avoid tying the tools too closely to SCons).
Initial doc content will include:
- writing the pipeline in "expanded" form (see
DM-33027), and including the resulting config and YAML files in rST pages via literalinclude;
- writing and rendering GraphViz diagrams for the full pipeline and each task, and including those in the rST pages with image directives.
There's a lot more we could do, but I'll create separate tickets for the rest. Everything described above has actually already been done on
DM-30891 branches; this ticket exists just to split that functionality out for separate review and merge.