Fix Version/s: None
Component/s: jenkins, pipelines_lsst_io
The purpose of this ticket is to modify the Jenkins pipelines to support builds of the pipelines.lsst.io documentation site using the new generation of documentation infrastructure in Documenteer 0.6 that is available through the conda-environment.
The documentation builds need to consider two cases:
- Development builds that act as CI for documentation contributions. That is, documentation builds should be added to the stack-os-matrix job.
- Documentation builds for releases and for maintenance of releases.
Documentation uploads are currently handled by a Jenkins job called "documenteer." I suggest that we retire this Jenkins job because it consumes the stack in a Docker container, making it inflexible especially for CIing development branches. (At the very least, the name "documenteer" for a Jenkins job should be retired because that's the name of a Python package containing Sphinx configuration and extensions, and so it's confusing for that name to be co-opted by a job that builds pipelines.lsst.io.)
Prerequisite tickets are:
DM-28324 will add LTD Conveyor to conda-forge, making it convenient to get the tooling to upload documentation builds.
DM-28326 will reconfigure the `pipelines_lsst_io` package to use the new configuration and build features from documenteer 0.6.
This is a description of how to run a documentation build. The prerequisite is that the stack is built and available on the Python path, that documenteer 0.6 is available, and that the pipelines_lsst_io package is cloned. This recipe assumes that the build is done from the root of the pipelines_lsst_io checkout, though this (likely) isn't necessary.
This is the build command:
For additional options, see https://documenteer.lsst.io/pipelines/stack-docs-cli.html though the defaults should be ok.
Documentation builds should likely be uploaded to LSST the Docs so that they're available for preview from a pipelines.lsst.io/v/ path. Documentation uploads are quite slow at the moment with a complete Doxygen sub-site, so it might be useful to make documentation uploads optional in the stack-os-matrix job. As a reminder, this work relies on
DM-28324 as a prerequisite.
The upload command is:
ltd upload --product pipelines --dir _build/html --git-ref <ref>
Additionally, two environment variables are needed and should be treated by secrets by Jenkins:
I can generate credentials for you.
The git ref is essentially the branch or tag being built and corresponds to the `/v/` paths, though `tickets/
DM-1234` forms are specially transformed to ` DM-1234` behind the scenes.
- `d_YYYY_MM_DD` daily eups tags
- `w_YYYY_NN` weekly eups tags
- `vNN_MM` major version tags
- `tickets/DM-NNNNN` single ticket branches
- Multiple ticket branches should probably be concatenated, like `tickets/DM-NNNNN-tickets/DM-MMMMM`. There was initially support for multiple git refs within LTD, but I think this will be dropped, so it's better if we treat our versions as a single string.
Versioning pipelines_lsst_io and release processes
A related aspect is that the `pipelines_lsst_io` package's versioning probably needs to be more carefully considered and included more rigorously in the release and release maintenance process. Right now, the `pipelines_lsst_io` package isn't tagged with releases, but a release branch is manually made as part of the release process (though the root of this release branch is potentially arbitrary, and in fact, it is arguable that releases should be made from `pipelines_lsst_io` master and release branches only made after a release is made for maintenance). This is all to say that versioning and release processes need to be thought-through because in the existing "documenteer" job the git ref of `pipelines_lsst_io` is manually specified and this has often caused confusion and inconsistencies.
The values of LTD_KEEPER_USER and LTD_KEEPER_PASSWORD and be migrated into LTD_USERNAME and LTD_PASSWORD. The AWS credentials are no longer used with LTD Conveyor.
Once I've finished
DM-28324 it'll be possible to include LTD Conveyor (i.e. the ltd upload command) directly in the native Python environment without needing to use a container to run the upload. You could still use the container pattern if you need it for isolation.
Just to update you: LTD Conveyor (i.e. the ltd upload tool) is now in conda-forge: https://github.com/conda-forge/ltd-conveyor-feedstock
My plan is to enable pipelines_lsst_io to be built under lsstsw, where building is creating the document site. This should make it easier to build this as part of Jenkins stack-os-matrix jobs as well as releases. This will include adding pipelines_lsst_io to repos.yaml, creating an eupspkg.cfg.sh to allow eupspkg to build the docs, and determining which directory doc upload should occur from.
The one problem with doing it this way is that we will build the docs on every platform (currently two), wasting resources, and also potentially upload from both platforms if uploading from stack-os-matrix jobs is desired.
So it looks like some surgery in ci-scripts and possibly extensive configuration changes in jenkins-dm-jobs are going to be necessary anyway.
Can you edit the eupspkg file to make the build a no-op on macOS IF running under CI (is there a Jenkins env var that is set?)
I've asked Tim to review overall, although I also asked for Adam Thornton to check the Jenkins groovy (which is mostly bash, actually).
Looks okay. I didn't spend too long looking at the groovy and I think I saw that multiple branch requests are all combined in the publish version.
I would like to modify the existing Jenkins jobs/pipelines as little as possible. In addition, we do not have the resources to develop completely new pipelines at any time in the near future, unless they're very simple (essentially: if you can specify the pipeline as, e.g., a shell script or GitHub action, we could translate that to Jenkins).
As I understand it, we would tack the stack-docs build command onto both the stack-os-matrix and the (nightly and weekly and official) release pipelines after the stack has been built.
The current documenteer pipeline, which I'm fine with retiring, not only uses a stack container but also an ltd-mason container that does its uploading. Are you proposing to replace the latter? That container already has (two sets of) credentials:
Do we need new ones, or can those continue to be used?