Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11216

Proof-of-concept for stack pipelines.lsst.io build

    XMLWordPrintable

    Details

      Description

      This ticket covers a demonstration of a pipelines.lsst.io build with content contributions from stack packages. We'll use the verify_metrics package since it's a data-only package that's easy to setup with EUPS (few dependencies).

      The build tool itself (build-stack-docs) is being developed on a separate ticket (DM-11004) in the documenteer package.

      For pipelines_lsst_io, this work will be merged into a DM-10635 integration branch to allow continued documentation deployments from master for the stack-less pipelines.lsst.io documentation builds on Travis.

      This work won't be deployed via CI.

        Attachments

          Issue Links

          is triggering

          Improvement - An improvement or enhancement to an existing feature or task. DM-13584 evaluate PoC for stack pipelines.lsst.io build

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11004 pipelines.lsst.io site build tool

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15024 Make package and module documentation directories exclusive

          • Done
          links to

          Web Link Community post: Using Python’s venv with newinstall’s Anaconda

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jsick Jonathan Sick added a comment - - edited

            Building pipelines.lsst.io with the Stack

            These steps allow you to reproduce a pipelines.lsst.io build integrated with Stack packages.

            The usual lsstsw-based build

            git clone https://github.com/lsst/lsstsw.git
            		 
            cd lsstsw
            		 
            ./bin/deploy
            		 
            source bin/setup.sh
            		 
            rebuild verify pipe_base
            		 
             # eups tags --clone bNNNN current

            I'm only building verify and pipe_base because those are packages that have new-style package documentation.

            Clone and install pipelines_lsst_io dependencies

            cd ..
            		 
            git clone https://github.com/lsst/pipelines_lsst_io
            		 
            cd pipelines_lsst_io
            		 
            git pull
            		 
            git checkout tickets/DM-11216
            		 
            pip install -r requirements.txt

            The tickets/DM-11216 branch is just the integration branch I'm using here. Ultimately we'd want to use lsstsw to clone the appropriate branch of pipelines_lsst_io for the requested set of Git refs.

            Can/should we make pipelines_lsst_io something that clone-able and install-able by lsstsw?

            Setup the EUPS dependencies of pipelines_lsst_io

            setup -r .

            pipelines_lsst_io has a ups/pipelines_lsst_io.table file that declares what packages it includes in its build.

            Build the integrated documentation

            build-stack-docs -d .

            The built HTML is in the _build/html/ directory of the cloned pipelines_lsst_io repo.

            The build-stack-docs tool symlinks the doc/ directories of setup EUPS packages into the pipelines_lsst_io repository. Then it runs Sphinx once over all the content (both from pipelines_lsst_io and individual packages).

            Next, we'd run a tool like ltd-mason (or really something new from github.com/lsst-sqre/ltd-conveyor) to upload the HTML to LSST the Docs.

            Show
            jsick Jonathan Sick added a comment - - edited Building pipelines.lsst.io with the Stack These steps allow you to reproduce a pipelines.lsst.io build integrated with Stack packages. The usual lsstsw-based build git clone https://github.com/lsst/lsstsw.git cd lsstsw ./bin/deploy source bin/setup.sh rebuild verify pipe_base # eups tags --clone bNNNN current I'm only building verify and pipe_base because those are packages that have new-style package documentation. Clone and install pipelines_lsst_io dependencies cd .. git clone https://github.com/lsst/pipelines_lsst_io cd pipelines_lsst_io git pull git checkout tickets/DM-11216 pip install -r requirements.txt The tickets/ DM-11216 branch is just the integration branch I'm using here. Ultimately we'd want to use lsstsw to clone the appropriate branch of pipelines_lsst_io for the requested set of Git refs. Can/should we make pipelines_lsst_io something that clone-able and install-able by lsstsw? Setup the EUPS dependencies of pipelines_lsst_io setup -r . pipelines_lsst_io has a ups/pipelines_lsst_io.table file that declares what packages it includes in its build. Build the integrated documentation build-stack-docs -d . The built HTML is in the _build/html/ directory of the cloned pipelines_lsst_io repo. The build-stack-docs tool symlinks the doc/ directories of setup EUPS packages into the pipelines_lsst_io repository. Then it runs Sphinx once over all the content (both from pipelines_lsst_io and individual packages). Next, we'd run a tool like ltd-mason (or really something new from github.com/lsst-sqre/ltd-conveyor) to upload the HTML to LSST the Docs.
            Hide
            Permalink
            jsick Jonathan Sick added a comment -

            In the linked community post, I've figured out how to install the documenteer tooling in an isolated Python virtual environment. This is great because it lets developers locally install and use documenteer without affecting their stack.

            Show
            jsick Jonathan Sick added a comment - In the linked community post , I've figured out how to install the documenteer tooling in an isolated Python virtual environment. This is great because it lets developers locally install and use documenteer without affecting their stack.
            Hide
            Permalink
            jsick Jonathan Sick added a comment -

            At long last, it's time to ship the new pipelines.lsst.io documentation system.

            Show
            jsick Jonathan Sick added a comment - At long last, it's time to ship the new pipelines.lsst.io documentation system. PR for pipelines_lsst_io itself:  https://github.com/lsst/pipelines_lsst_io/pull/99 PR for the Jenkins job:  https://github.com/lsst-sqre/jenkins-dm-jobs/pull/479
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            These changes look great. I'm assuming you don't need me to also review the documenteer changes that enable this.

            Show
            krughoff Simon Krughoff added a comment - These changes look great. I'm assuming you don't need me to also review the documenteer changes that enable this.
            Hide
            Permalink
            krughoff Simon Krughoff added a comment -

            I see there is also a branch for this ticket in verify_metrics. Is that something I should look at?

            Show
            krughoff Simon Krughoff added a comment - I see there is also a branch for this ticket in verify_metrics . Is that something I should look at?
            Hide
            Permalink
            jsick Jonathan Sick added a comment -

            Thanks Simon Krughoff, this is good. The Documenteer stuff was all on separate tickets anyhow.

            I completely forgot I also had the prototyping branch with verify_metrics. I'll take a look and see if that needs to be updated.

            Show
            jsick Jonathan Sick added a comment - Thanks Simon Krughoff , this is good. The Documenteer stuff was all on separate tickets anyhow. I completely forgot I also had the prototyping branch with verify_metrics . I'll take a look and see if that needs to be updated.
            Hide
            Permalink
            jsick Jonathan Sick added a comment -

            It looks like this ticket branch in verify_metrics got superseded by other work that added a documentation directory, so there's no need to merge tickets/DM-11216 in verify_metrics.

            Show
            jsick Jonathan Sick added a comment - It looks like this ticket branch in verify_metrics got superseded by other work that added a documentation directory, so there's no need to merge tickets/ DM-11216 in verify_metrics .

              People

              Assignee:
              jsick Jonathan Sick
              Reporter:
              jsick Jonathan Sick
              Reviewers:
              Simon Krughoff
              Watchers:
              John Swinbank, Jonathan Sick, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.