The purpose of this ticket is to modify the Jenkins pipelines to support builds of the pipelines.lsst.io documentation site using the new generation of documentation infrastructure in Documenteer 0.6 that is available through the conda-environment.

The documentation builds need to consider two cases:

1. Development builds that act as CI for documentation contributions. That is, documentation builds should be added to the stack-os-matrix job.
2. Documentation builds for releases and for maintenance of releases.

Documentation uploads are currently handled by a Jenkins job called "documenteer." I suggest that we retire this Jenkins job because it consumes the stack in a Docker container, making it inflexible especially for CIing development branches. (At the very least, the name "documenteer" for a Jenkins job should be retired because that's the name of a Python package containing Sphinx configuration and extensions, and so it's confusing for that name to be co-opted by a job that builds pipelines.lsst.io.)

 Prerequisite tickets are:

1. DM-28324 will add LTD Conveyor to conda-forge, making it convenient to get the tooling to upload documentation builds.

2. DM-28326 will reconfigure the `pipelines_lsst_io` package to use the new configuration and build features from documenteer 0.6.

Build recipe

This is a description of how to run a documentation build. The prerequisite is that the stack is built and available on the Python path, that documenteer 0.6 is available, and that the pipelines_lsst_io package is cloned. This recipe assumes that the build is done from the root of the pipelines_lsst_io checkout, though this (likely) isn't necessary.

This is the build command:

stack-docs build

For additional options, see https://documenteer.lsst.io/pipelines/stack-docs-cli.html though the defaults should be ok.

Upload recipe

Documentation builds should likely be uploaded to LSST the Docs so that they're available for preview from a pipelines.lsst.io/v/ path. Documentation uploads are quite slow at the moment with a complete Doxygen sub-site, so it might be useful to make documentation uploads optional in the stack-os-matrix job. As a reminder, this work relies on DM-28324 as a prerequisite.

The upload command is:

ltd upload --product pipelines --dir _build/html --git-ref <ref>

Additionally, two environment variables are needed and should be treated by secrets by Jenkins:

• LTD_USERNAME
• LTD_PASSWORD

I can generate credentials for you.

The git ref is essentially the branch or tag being built and corresponds to the `/v/` paths, though `tickets/DM-1234` forms are specially transformed to `DM-1234` behind the scenes.

• `d_YYYY_MM_DD` daily eups tags

• `w_YYYY_NN` weekly eups tags

• `vNN_MM` major version tags

• `tickets/DM-NNNNN` single ticket branches

• Multiple ticket branches should probably be concatenated, like `tickets/DM-NNNNN-tickets/DM-MMMMM`. There was initially support for multiple git refs within LTD, but I think this will be dropped, so it's better if we treat our versions as a single string. 

Versioning pipelines_lsst_io and release processes

A related aspect is that the `pipelines_lsst_io` package's versioning probably needs to be more carefully considered and included more rigorously in the release and release maintenance process. Right now, the `pipelines_lsst_io` package isn't tagged with releases, but a release branch is manually made as part of the release process (though the root of this release branch is potentially arbitrary, and in fact, it is arguable that releases should be made from `pipelines_lsst_io` master and release branches only made after a release is made for maintenance). This is all to say that versioning and release processes need to be thought-through because in the existing "documenteer" job the git ref of `pipelines_lsst_io` is manually specified and this has often caused confusion and inconsistencies.