Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-733

Add Documenteer to the Conda environment (scipipe_conda_env)

    XMLWordPrintable

Details

    • RFC
    • Status: Implemented
    • Resolution: Done
    • DM
    • None

    Description

      I would like to add Documenteer, specifically the lsst-documenteer-pipelines meta-package (available on conda-forge), to the Conda environment for the stack (scipipe_conda_env). This change will provide all of our documentation dependencies in the default Python environment for both developers and CI systems. This change will also deprecate our use of a requirements.txt file in the pipelines_lsst_io repository for setting the version of Documenteer that should be used for CI and local builds of stack documentation.

      Concurrently, I would like to upgrade the version of Documenteer that we are using for production to 0.6.2 (from 0.5.6). As an immediate effect, this will upgrade us to the current versions of packages in our documentation ecosystem, such as Sphinx (1.7 to 3.2), Numpydoc, and sphinx-automodapi. This upgrade will also immediately improve the developer experience by solving a critical bug related to "recursion errors" in single-package documentation builds by changing how Documenteer uses the Sphinx configuration system. See the Documenteer release notes for details.

      Through this RFC, the final output of the documentation will not change in any outwardly-visible ways.

      Future benefits

      This proposal paves the way for future improvements (though they are not in the scope of this RFC's implementation):

      • Sphinx documentation builds could be run as part of the stack-os-matrix Jenkins job. Doing so would help us to validate docstrings and reStructuredText and even push draft documentation versions as part of ticket branch CI to improve the efficiency of documentation reviews.
      • Documenteer 0.6 includes the capability to configure and run a Doxygen build, embed that Doxygen content into the Sphinx documentation, and cross-reference from Sphinx documentation into the Doxygen-based C++ API reference. This capability is disabled in our current usage but can be turned on following a future RFC.
      • Take advantage of new Sphinx features (such as support for incorporating type annotations in Python API documentation).

      Tasks to implement this RFC

      1. Add lsst-documenteer-pipelines to scipipe_conda_env.
      2. Update the pipelines_lsst_io repository. First, change the conf.py file to use the new configuration API as outlined in https://documenteer.lsst.io/pipelines/configuration.html. Second, remove the requirements.txt file.
      3. Update the documenteer job in Jenkins so that it uses the Documenteer installed in the conda environment rather than installing an additional (and conflicting) copy of documenteer through pip.
      4. Update the conf.py files of packages (doc/conf.py) to use the new configuration API. We have endeavoured to maintain backwards compatibility in the new version of Documenteer with the old configuration API, but there's still a great benefit to upgrading packages as quickly and uniformly as possible.
      5. Update documentation (primarily in the Developer Guide) and templates to reflect the new conf.py files and the fact that Documenteer is now included in the Conda environment.

      Timing

      The 21.0.0 release of the LSST Science Pipelines is expected in November, and I am happy to work around this release, either by implementing the work now or delaying until after the next major release.

      Attachments

        Issue Links

          Activity

            ktl Kian-Tat Lim added a comment -

            The one hesitation I have is that, to my understanding, documenteer is entirely a build-time dependency, not a use-time dependency. We would like to have only use-time dependencies in the base scipipe_conda_env (to become the rubin-env metapackage) environment. I have no objections to including it in a build-time/developer variation of that environment, however.

            ktl Kian-Tat Lim added a comment - The one hesitation I have is that, to my understanding, documenteer is entirely a build-time dependency, not a use-time dependency. We would like to have only use-time dependencies in the base scipipe_conda_env (to become the rubin-env metapackage) environment. I have no objections to including it in a build-time/developer variation of that environment, however.

            I was also concerned about this, and certainly, I agree that separating development and runtime dependencies is important to the final user experience (we do this for Documenteer itself, for example).

            When do we expect to have a build-time/development conda environment ready to use? Are there some related tickets to track?

            jsick Jonathan Sick added a comment - I was also concerned about this, and certainly, I agree that separating development and runtime dependencies is important to the final user experience (we do this for Documenteer itself, for example). When do we expect to have a build-time/development conda environment ready to use? Are there some related tickets to track?
            Parejkoj John Parejko added a comment -

            I very much support this, as someone who builds docs locally to check them out.

            To the use-time vs. build-time question: we distribute compilers, so how is this different?

            Parejkoj John Parejko added a comment - I very much support this, as someone who builds docs locally to check them out. To the use-time vs. build-time question: we distribute compilers, so how is this different?
            tjenness Tim Jenness added a comment -

            I'm strongly in favor of this.

            tjenness Tim Jenness added a comment - I'm strongly in favor of this.

            It looks like there's overall support. I'll suggest that the consensus is that we should go ahead with this as an addition to scipipe_conda_env as suggested, and then once we have the infrastructure to have separate build and runtime dependencies, we can move documenteer over to the build-only Conda environment.

            The last question is whether we want to tackle this now (before version 21.0 is released or not). Do you have any preference gcomoretto? Implementing this work will require adjustments to the Jenkins jobs, so the question is whether that will interfere with release preparations or not. I also likely need assistance in modifying the Jenkins jobs, so coordinating the availability of a Jenkins expert during this work would be great.

            jsick Jonathan Sick added a comment - It looks like there's overall support. I'll suggest that the consensus is that we should go ahead with this as an addition to scipipe_conda_env as suggested, and then once we have the infrastructure to have separate build and runtime dependencies, we can move documenteer over to the build-only Conda environment. The last question is whether we want to tackle this now (before version 21.0 is released or not). Do you have any preference gcomoretto ? Implementing this work will require adjustments to the Jenkins jobs, so the question is whether that will interfere with release preparations or not. I also likely need assistance in modifying the Jenkins jobs, so coordinating the availability of a Jenkins expert during this work would be great.

            Since this doesn't seem a small change only to one package, I suggest we postpone it after the 21.0.0 release.

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - Since this doesn't seem a small change only to one package, I suggest we postpone it after the 21.0.0 release.
            tjenness Tim Jenness added a comment -

            jsick can you please adopt this RFC assuming you still want it done? Also attach a DM ticket and I imagine you would assign bvan.

            tjenness Tim Jenness added a comment - jsick can you please adopt this RFC assuming you still want it done? Also attach a DM ticket and I imagine you would assign bvan .
            ktl Kian-Tat Lim added a comment -

            Actually, we should avoid pinging Brian; you can assign the ticket(s), including updating the environment and assisting with modifying Jenkins, to me.

            ktl Kian-Tat Lim added a comment - Actually, we should avoid pinging Brian; you can assign the ticket(s), including updating the environment and assisting with modifying Jenkins, to me.

            Yes, thanks. I'm working on the tickets for implementation. Thanks for volunteering to help with the hard bits of the Jenkins updates ktl.

            jsick Jonathan Sick added a comment - Yes, thanks. I'm working on the tickets for implementation. Thanks for volunteering to help with the hard bits of the Jenkins updates ktl .

            documenteer has already been added to rubin-env metapackage - so adopting the metapackage would implement the RFC.

            bvan Brian Van Klaveren added a comment - documenteer has already been added to rubin-env metapackage - so adopting the metapackage would implement the RFC.
            ktl Kian-Tat Lim added a comment - - edited

            While DM-26671 is about using rubin-env rather than specifically adding documenteer, documenteer has already been added to the environment in https://github.com/conda-forge/rubinenv-feedstock/pull/7. We'll therefore use DM-26671 as a triggered implementation ticket.

            If there are additional tickets for Jenkins updates etc., please link them.

            ktl Kian-Tat Lim added a comment - - edited While DM-26671 is about using rubin-env rather than specifically adding documenteer, documenteer has already been added to the environment in https://github.com/conda-forge/rubinenv-feedstock/pull/7 . We'll therefore use DM-26671 as a triggered implementation ticket. If there are additional tickets for Jenkins updates etc., please link them.
            ktl Kian-Tat Lim added a comment -

            Apologies for not noticing that the addition to rubin-env was done incorrectly. (I looked at the title and not the content of the RFC.) This will be fixed in rubin-env 0.2.0.

            ktl Kian-Tat Lim added a comment - Apologies for not noticing that the addition to rubin-env was done incorrectly. (I looked at the title and not the content of the RFC.) This will be fixed in rubin-env 0.2.0.

            Hm.  May have pushed the button prematurely.  While the title action has definitely been done (long ago), some of the follow-ups have not.  Nevertheless, I think they may belong in a different RFC (or not even require one).

            ktl Kian-Tat Lim added a comment - Hm.  May have pushed the button prematurely.  While the title action has definitely been done (long ago), some of the follow-ups have not.  Nevertheless, I think they may belong in a different RFC (or not even require one).
            tjenness Tim Jenness added a comment -

            Do you want to remove DM-28328 from this RFC then?

            tjenness Tim Jenness added a comment - Do you want to remove DM-28328 from this RFC then?
            ktl Kian-Tat Lim added a comment -

            I'll leave that for the owner (as I should have done for the overall RFC status).

            ktl Kian-Tat Lim added a comment - I'll leave that for the owner (as I should have done for the overall RFC status).

            People

              jsick Jonathan Sick
              jsick Jonathan Sick
              Brian Van Klaveren, Gabriele Comoretto [X] (Inactive), John Parejko, Jonathan Sick, Kian-Tat Lim, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.