Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28328

Add documentation builds to stack-os-matrix

    XMLWordPrintable

Details

    • Story
    • Status: Reviewed
    • Resolution: Unresolved
    • None
    • jenkins, pipelines_lsst_io
    • None
    • Architecture
    • No

    Description

      The purpose of this ticket is to modify the Jenkins pipelines to support builds of the pipelines.lsst.io documentation site using the new generation of documentation infrastructure in Documenteer 0.6 that is available through the conda-environment.

      The documentation builds need to consider two cases:

      1. Development builds that act as CI for documentation contributions. That is, documentation builds should be added to the stack-os-matrix job.
      2. Documentation builds for releases and for maintenance of releases.

      Documentation uploads are currently handled by a Jenkins job called "documenteer." I suggest that we retire this Jenkins job because it consumes the stack in a Docker container, making it inflexible especially for CIing development branches. (At the very least, the name "documenteer" for a Jenkins job should be retired because that's the name of a Python package containing Sphinx configuration and extensions, and so it's confusing for that name to be co-opted by a job that builds pipelines.lsst.io.)

       Prerequisite tickets are:

      1. DM-28324 will add LTD Conveyor to conda-forge, making it convenient to get the tooling to upload documentation builds.

      2. DM-28326 will reconfigure the `pipelines_lsst_io` package to use the new configuration and build features from documenteer 0.6.

       

      Build recipe

       

      This is a description of how to run a documentation build. The prerequisite is that the stack is built and available on the Python path, that documenteer 0.6 is available, and that the pipelines_lsst_io package is cloned. This recipe assumes that the build is done from the root of the pipelines_lsst_io checkout, though this (likely) isn't necessary.

      This is the build command:

      stack-docs build

      For additional options, see https://documenteer.lsst.io/pipelines/stack-docs-cli.html though the defaults should be ok.

      Upload recipe

      Documentation builds should likely be uploaded to LSST the Docs so that they're available for preview from a pipelines.lsst.io/v/ path. Documentation uploads are quite slow at the moment with a complete Doxygen sub-site, so it might be useful to make documentation uploads optional in the stack-os-matrix job. As a reminder, this work relies on DM-28324 as a prerequisite.

      The upload command is:

      ltd upload --product pipelines --dir _build/html --git-ref <ref>

      Additionally, two environment variables are needed and should be treated by secrets by Jenkins:

      • LTD_USERNAME
      • LTD_PASSWORD

      I can generate credentials for you.

      The git ref is essentially the branch or tag being built and corresponds to the `/v/` paths, though `tickets/DM-1234` forms are specially transformed to `DM-1234` behind the scenes.

      • `d_YYYY_MM_DD` daily eups tags
      • `w_YYYY_NN` weekly eups tags
      • `vNN_MM` major version tags
      • `tickets/DM-NNNNN` single ticket branches
      • Multiple ticket branches should probably be concatenated, like `tickets/DM-NNNNN-tickets/DM-MMMMM`. There was initially support for multiple git refs within LTD, but I think this will be dropped, so it's better if we treat our versions as a single string. 

      Versioning pipelines_lsst_io and release processes

      A related aspect is that the `pipelines_lsst_io` package's versioning probably needs to be more carefully considered and included more rigorously in the release and release maintenance process. Right now, the `pipelines_lsst_io` package isn't tagged with releases, but a release branch is manually made as part of the release process (though the root of this release branch is potentially arbitrary, and in fact, it is arguable that releases should be made from `pipelines_lsst_io` master and release branches only made after a release is made for maintenance). This is all to say that versioning and release processes need to be thought-through because in the existing "documenteer" job the git ref of `pipelines_lsst_io` is manually specified and this has often caused confusion and inconsistencies.

      Attachments

        Issue Links

          Activity

            ktl Kian-Tat Lim added a comment -

            The one problem with doing it this way is that we will build the docs on every platform (currently two), wasting resources, and also potentially upload from both platforms if uploading from stack-os-matrix jobs is desired.

            So it looks like some surgery in ci-scripts and possibly extensive configuration changes in jenkins-dm-jobs are going to be necessary anyway.

            ktl Kian-Tat Lim added a comment - The one problem with doing it this way is that we will build the docs on every platform (currently two), wasting resources, and also potentially upload from both platforms if uploading from stack-os-matrix jobs is desired. So it looks like some surgery in ci-scripts and possibly extensive configuration changes in jenkins-dm-jobs are going to be necessary anyway.
            tjenness Tim Jenness added a comment -

            Can you edit the eupspkg file to make the build a no-op on macOS IF running under CI (is there a Jenkins env var that is set?)

            tjenness Tim Jenness added a comment - Can you edit the eupspkg file to make the build a no-op on macOS IF running under CI (is there a Jenkins env var that is set?)
            ktl Kian-Tat Lim added a comment -

            I've asked Tim to review overall, although I also asked for athornton to check the Jenkins groovy (which is mostly bash, actually).

            ktl Kian-Tat Lim added a comment - I've asked Tim to review overall, although I also asked for athornton to check the Jenkins groovy (which is mostly bash, actually).
            tjenness Tim Jenness added a comment -

            Looks okay. I didn't spend too long looking at the groovy and I think I saw that multiple branch requests are all combined in the publish version.

            tjenness Tim Jenness added a comment - Looks okay. I didn't spend too long looking at the groovy and I think I saw that multiple branch requests are all combined in the publish version.
            tjenness Tim Jenness added a comment -

            What's the status of this ticket? It was reviewed but not merged.

            tjenness Tim Jenness added a comment - What's the status of this ticket? It was reviewed but not merged.

            People

              ktl Kian-Tat Lim
              jsick Jonathan Sick
              Tim Jenness
              Jonathan Sick, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Jenkins

                  No builds found.