Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9984

Jupyterlab-based stack environment prototype

    Details

    • Epic Name:
      sqre-s17-jupyterlab
    • Story Points:
      30
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Spring 2017

      Description

      The goal is to build experience with JupyterLab in SQuaRE by building a prototype service that can execute stack command-line tasks (or supertasks, if available). Since this is a learning activity outcomes are loosely defined but at the minimum include a configuration-managed deployable service.

        Attachments

          Issue Links

            Activity

            Hide
            frossie Frossie Economou added a comment -

            Adam Thornton since this is exploratory work you may self-merge all PRs unless you are specifically looking for input.

            Show
            frossie Frossie Economou added a comment - Adam Thornton since this is exploratory work you may self-merge all PRs unless you are specifically looking for input.
            Hide
            athornton Adam Thornton added a comment -

            Docker image lsstsqre/jupyterlabdemo:0.0.6
            Github lsst-sqre/jupyterlabdemo commit c80dda68f0a13c5ac54f5d7eff38cfab14a3230e

            Implements a Jupyterhub spawner (no authentication) for a Jupyterlab environment; both Python2 and Python3 kernels contain an LSST stack environment.

            Eventually we will want k8s to run either the lab/hub combo or for hub to spawn new containers (which requires scary privilege on the hub), and to run as a k8s service. Then anywhere we have a cluster we can just drop in these things.

            Show
            athornton Adam Thornton added a comment - Docker image lsstsqre/jupyterlabdemo:0.0.6 Github lsst-sqre/jupyterlabdemo commit c80dda68f0a13c5ac54f5d7eff38cfab14a3230e Implements a Jupyterhub spawner (no authentication) for a Jupyterlab environment; both Python2 and Python3 kernels contain an LSST stack environment. Eventually we will want k8s to run either the lab/hub combo or for hub to spawn new containers (which requires scary privilege on the hub), and to run as a k8s service. Then anywhere we have a cluster we can just drop in these things.
            Hide
            athornton Adam Thornton added a comment -

            We run hub and spawn lab containers. Next step: persistent storage.

            Show
            athornton Adam Thornton added a comment - We run hub and spawn lab containers. Next step: persistent storage.
            Hide
            frossie Frossie Economou added a comment -

            Awesomeness.

            This was a timebox epic in which we got through more than expected.

            This epic delivered:

            • An experimental sandbox at jupyterlabdemo.lsst.codes (not a stable service; frequently redeployed with no notice; functionality subject to change).
            • A Dockerfile for installing JupyterHub and JupyterLab components onto our LSST stack container and exposing the stack kernel to Jupyter.
            • A Kubernetes configuration to deploy this service on the GKE cluster.
            • Our deployment involves JupyterHub spawning Jupyterlabs in individual pods based on a SQuaRE’s CI-built Docker container dinstribution of the stack.
            • Authentication to this service via SQuaRE’s usual “shim” authentication mechanism that uses Github organization membership for OAuth authentication
            • Consistent Unix uid/gid mapping for users using Github as a shim (your user id is your github id, your groups are the orgs you belong to).

            Code for this epic can be found in the following repositories:

            https://github.com/lsst-sqre/jupyterlabdemo :
            (JupyterLab container provisioning and Kubernetes cofig)
            https://github.com/lsst-sqre/k8s-jupyterlabdemo-nginx :
            (proxy and Kubernetes config)
            https://github.com/lsst-sqre/ghowlauth :
            (authenticator)

            Also, a great notebook was contributed by Simon Krughoff (attached to the next epic) that has been invaluable in testing.

            Show
            frossie Frossie Economou added a comment - Awesomeness. This was a timebox epic in which we got through more than expected. This epic delivered: An experimental sandbox at jupyterlabdemo.lsst.codes ( not a stable service; frequently redeployed with no notice; functionality subject to change). A Dockerfile for installing JupyterHub and JupyterLab components onto our LSST stack container and exposing the stack kernel to Jupyter. A Kubernetes configuration to deploy this service on the GKE cluster. Our deployment involves JupyterHub spawning Jupyterlabs in individual pods based on a SQuaRE’s CI-built Docker container dinstribution of the stack. Authentication to this service via SQuaRE’s usual “shim” authentication mechanism that uses Github organization membership for OAuth authentication Consistent Unix uid/gid mapping for users using Github as a shim (your user id is your github id, your groups are the orgs you belong to). Code for this epic can be found in the following repositories: https://github.com/lsst-sqre/jupyterlabdemo : (JupyterLab container provisioning and Kubernetes cofig) https://github.com/lsst-sqre/k8s-jupyterlabdemo-nginx : (proxy and Kubernetes config) https://github.com/lsst-sqre/ghowlauth : (authenticator) Also, a great notebook was contributed by Simon Krughoff (attached to the next epic) that has been invaluable in testing.

              People

              • Assignee:
                athornton Adam Thornton
                Reporter:
                frossie Frossie Economou
                Reviewers:
                Frossie Economou
                Watchers:
                Adam Thornton, Frossie Economou, Gregory Dubois-Felsmann, Jonathan Sick
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel