I would like to add Documenteer, specifically the lsst-documenteer-pipelines meta-package (available on conda-forge), to the Conda environment for the stack (scipipe_conda_env). This change will provide all of our documentation dependencies in the default Python environment for both developers and CI systems. This change will also deprecate our use of a requirements.txt file in the pipelines_lsst_io repository for setting the version of Documenteer that should be used for CI and local builds of stack documentation.

Concurrently, I would like to upgrade the version of Documenteer that we are using for production to 0.6.2 (from 0.5.6). As an immediate effect, this will upgrade us to the current versions of packages in our documentation ecosystem, such as Sphinx (1.7 to 3.2), Numpydoc, and sphinx-automodapi. This upgrade will also immediately improve the developer experience by solving a critical bug related to "recursion errors" in single-package documentation builds by changing how Documenteer uses the Sphinx configuration system. See the Documenteer release notes for details.

Through this RFC, the final output of the documentation will not change in any outwardly-visible ways.

Future benefits

This proposal paves the way for future improvements (though they are not in the scope of this RFC's implementation):

• Sphinx documentation builds could be run as part of the stack-os-matrix Jenkins job. Doing so would help us to validate docstrings and reStructuredText and even push draft documentation versions as part of ticket branch CI to improve the efficiency of documentation reviews.
• Documenteer 0.6 includes the capability to configure and run a Doxygen build, embed that Doxygen content into the Sphinx documentation, and cross-reference from Sphinx documentation into the Doxygen-based C++ API reference. This capability is disabled in our current usage but can be turned on following a future RFC.
• Take advantage of new Sphinx features (such as support for incorporating type annotations in Python API documentation).

Tasks to implement this RFC

1. Add lsst-documenteer-pipelines to scipipe_conda_env.
2. Update the pipelines_lsst_io repository. First, change the conf.py file to use the new configuration API as outlined in https://documenteer.lsst.io/pipelines/configuration.html. Second, remove the requirements.txt file.
3. Update the documenteer job in Jenkins so that it uses the Documenteer installed in the conda environment rather than installing an additional (and conflicting) copy of documenteer through pip.
4. Update the conf.py files of packages (doc/conf.py) to use the new configuration API. We have endeavoured to maintain backwards compatibility in the new version of Documenteer with the old configuration API, but there's still a great benefit to upgrading packages as quickly and uniformly as possible.
5. Update documentation (primarily in the Developer Guide) and templates to reflect the new conf.py files and the fact that Documenteer is now included in the Conda environment.

Timing

The 21.0.0 release of the LSST Science Pipelines is expected in November, and I am happy to work around this release, either by implementing the work now or delaying until after the next major release.