So far, I have done the following:

• Included a small test gen3 repository with the repo, to enable local development (with daf_butler installed).
• Made a Dockerfile that enables running the panel server in a container

Leanne Guy and Colin Slater have tested using this docker image and will contact folks at NCSA regarding how it and similar custom servers could be hosted in a secure and sanctioned way for the benefit of developers.

Spoke to Michelle, she will look at getting resources to host this service

I spoke to Greg Daues and Michelle Butler [X] about this today. Sounds like this will be deployed on our k8s-dev cluster.

For now we will need:

1. Requested namespace name for the service
2. List of users that will need access to the namespace. Please note if any of these users should have read only access to the namespace. Otherwise, I will assume they should be admins with full access to it.

Additional discussion points from today regarding requirements for the service:

1. Since this service will live in a k8s cluster at NCSA, it will be subject to the same IP whitelist / VPN requirement as the RSPs. This is a technical requirement since the whole cluster is behind the same ingress controller and firewalls.

2. Required level of admin involvement. It's a matter of minutes to give people access to the cluster and start deploying pods in their namespace. Guiding people through the entire service deployment is a bigger job, and should be accounted for.

3. Whether or not we have fancy storage or networking requirements to deal with. IE: is this service simply reading input data and presenting it, or does it require its own persistent backend storage?

Thanks Matthew!

1. I don't have any great ideas for this. Something including "butler-plot-browser" or something?
2. Should include at least me, Lauren MacArthur Sophie Reed Yusra AlSayyad Lee Kelvin Colin Slater, and eventually more, I'm sure. OK for all to be admin.

Additional points:
1. Fine.
2. OK. Presumably at least one pipelines developer (me) and possibly more would probably be interested in how the service deployment works.
3. This particular service is simply reading data from /project and presenting it. I could imagine future services that might possibly requiring some backend storage, but no need to set things up for that now.

Some questions from me:
a) How will the development cycle work with this? If I change the dashboard code on github, will it automatically be updated when a user next "launches a pod" (is that the right terminology?) or will I (or someone) have to do something on the admin end to update?

b) Once this is done, will it be possible to easily deploy another different dashboard in this same way? I can imagine that there could be several use cases for simple (and more complex) tools like this for pipeline developers.

Hi Matt, thanks for jumping on this so quickly. I have a few things about the deployment plan I'd like to work out on the pipelines side before we dive into configuring kubernetes, so I think we can give you a more complete picture in a little bit.

I'm wondering if we have a deployment model for things on the K8 environment as a whole ??   Do we let Frossie Economou team know that there is a new app going on the K8? and it shoudl be tested on int first..etc?   is this somthing that developers deploy on their own?   or do they work with NCSA?  or do they work with SQRE folks on teh deployment?   etc?   

M. 

> Since this service will live in a k8s cluster at NCSA, it will be subject to the same IP whitelist / VPN requirement as the RSPs. This is a technical requirement since the whole cluster is behind the same ingress controller and firewalls.

I'd like this service to not be behind a firewall. It will be a read-only service as Tim Morton [X] explains where people will browse and possibly download plots. Could we set it up as per squash.lsst.codes with login using github credentials?

After reviewing the resources, this  (K8) is the only resource that NCSA has for running docker images.   

M. 

For what it's worth, the specific proximal motivation for development of the current app under discussion is for plots to be able to be easily browsed from an iPad (a use case often required by Lauren MacArthur, and perhaps others). I now understand that being behind the firewall would prevent that specific use case, so we do need to find an alternative solution.

I have successfully experimented with deploying this in a kubernetes container with minikube on my laptop, using the following deployment: https://github.com/timothydmorton/pipe-analysis-navigator/tree/main/kube. Still to-do: mount volumes so that data outside the container can be accessed.

To summarize my understanding of the current state here: it would be preferable to have this deployed outside the firewall, and that should be an eventual goal, but it would also be immediately useful to Yusra AlSayyad, Sophie Reed, and perhaps others on the DRP team to have it up anywhere (even behind firewall at NCSA) sooner. Michelle Butler [X], with whom should I coordinate to get a deployment of this tool up at NCSA?

FYI, for any interested party who wants to use this before a stable deployment is up, I currently have a server running at port 56789 on lsst-devl. Obviously not a permanent solution, but feel free to tunnel in and use to browse gen3 analysis plots.

Tim Morton [X] would you please comment on this linked issue to detail which directories the service will need access to? Also, Michelle Butler [X] would like to know if this app requires "Butler middleware" or not.

Regarding the "gen3 middleware" question, I think the answer is that the app only requires daf_butler as a standalone package, and nothing else in the LSST software stack. If this doesn't answer the question Michelle Butler [X], I'm happy to clarify further.

I created a single-node Kubernetes cluster on the NCSA Radiant system using k3s to provide a platform for running PAN (Pipe Analysis Navigator) with public access. The evolving documentation and code repository driving the deployment are linked on https://lsst-sandbox.ncsa.illinois.edu/. Tim Morton [X] and I identified a few bugs in the container image that are preventing the service from working. We will coordinate to determine how to proceed.

GREAT!  are we mounting the data area from the LSST GPFS?  Were you able to get that done?   do I need to push on something?  let me know. 

Michelle Butler [X], we are waiting on https://jira.lsstcorp.org/browse/IHS-4663 to be resolved. It was assigned to JD, who I originally spoke to along with Matt Long and who specifically directed me to open the IHS Jira issue on https://jira.lsstcorp.org/ in the first place. "Fortunately" there are some bugs that need to be squashed to get the service running with the local test data before it becomes a blocker.

The latest commit yields a functioning service at https://lsst-sandbox.ncsa.illinois.edu/pipe-analysis-navigator/. We are in standby until the GPFS access issue is resolved.

In the meantime we can research how to use Dask Kubernetes.

Andrew Manning [X] this is fantastic, thanks!  Super exciting to see this running, and as a prototype for simple apps like this.  And for everyone else, to clarify the dask comment, I just pointed Andrew to dask-kubernetes as something we will need in the future if (when) we want an app like this to be able to use dask in the backend (e.g. to make interactive plots with lots of data).  So not immediately urgent, but definitely will be required sometime in the near-ish future.

Also, barring any better suggestions, I'll plan to rename this to "analysis-plot-navigator".  Lauren MacArthur Sophie Reed Yusra AlSayyad, does this sound alright to you guys?  Or perhaps "analysis-drp-navigator"?  Preferences?

This looks really good Andrew Manning [X], Michelle Butler [X] - thank you for setting this up!

Today I installed ArgoCD and, after the discussion Tim Morton [X], Colin Slater and I had about initial goals for developer usability, I configured ArgoCD in the following way:

Members of the lsst-dm GitHub organization can log in to the ArgoCD web GUI at https://lsst-sandbox.ncsa.illinois.edu/argo-cd/ using the GitHub login button. They can create/modify/delete applications in the lsst-dm ArgoCD project, where they must select a target namespace that has the prefix lsst-dm-. For example, logging in via GitHub (i.e., as a non-admin user), I was able to create an app based on the git repo https://github.com/lsst-dm/pipe-analysis-navigator, which is an import of Tim Morton [X]'s original repo at https://github.com/timothydmorton/pipe-analysis-navigator but refactored to include a Helm Chart definition and a GitHub workflow definition. When commits are made to the main branch, a GitHub Action builds and pushes the lsstdm/pipe-analysis-navigator Docker image. The ArgoCD interface allows us to see the updates deployed almost instantly. The Helm charts that define an ArgoCD app can include anything; in this example, it defines an Ingress so that the service is available at https://lsst-sandbox.ncsa.illinois.edu/pan-test/.

This is clearly a powerful platform and several aspects of this system are new to me, so for the moment I would like to restrict access to only those individuals that specifically ask me for permission to use the cluster.

FYI, we [SQuaRE] do not allow ArgoCD to automatically sync production deployments. Obviously your usecase might be different.

Frossie Economou that is definitely a prudent policy. I am calling this a "sandbox" for a reason; it will be easy for developers to try out ideas but also easy for a nearby toddler to wreck everything they've made Coincidentally, excitement about this concept spiraled out quickly beyond my original impression that this was for the singular purpose of the PAN. Although it currently stands on relatively shaky foundations, it should be simple to migrate to a more robust infrastructure when desired.

We added another powerful feature today: Vault. This page describes how to use Vault to store secrets securely, and how to then use those secrets in app Helm charts in ArgoCD. This method ensures that secrets are never stored unencrypted outside the Kubernetes cluster itself, and deployment manifests can define Secrets by referencing a Vault path instead of including the sensitive info itself.

Tim Morton [X] I spoke to Michelle Gower today to ask her about these error messages like

Failed to load Butler from /project/hsc/gen3repo/rc2w22_ssw25/butler.yaml 

Failed to import cls 'lsst.daf.butler.datastores.posixDatastore.PosixDatastore' for config 

My understanding is that we need to be executing the panel serve command in a proper environment, which means we will probably need to switch to using the lsstsqre/centos:d_latest Docker image for the container (but with the additional holoviz and bokeh dependencies installed in a derived Dockerfile).

There is another issue she mentioned, however, which is that some of those Butler repos are only expected to work for certain versions of the stack, so you will probably need to include more error catching blocks in https://github.com/lsst-dm/pipetask-plot-navigator/blob/main/src/dashboard_gen3.py or do some tests on the user-selected repo prior to querying it for information to display.

That problem is that you are using an ancient datastore that was never released and which I finally deleted yesterday (DM-29354).

Replace posixDatastore.PosixDatastore with fileDatastore.FileDatastore if you haven't already.

That is probably for the app developer Tim Morton [X]..  above from Tim Jenness ?? correct?  

I did not reply because I thought I was atted. I replied because I saw that import failure passing by and was worried people would not know how to fix it immediately.

Hi all- Yes, indeed, the drop-down selector in the app shows all butlers in the root dir, but older ones won't load because they're old.  However, it should work for, e.g. /project/hsc/gen3repo/rc2w02_ssw03.

Regarding the environment, I was hoping that just the daf_butler installation would be sufficient, and that we wouldn't need a full stack environment in order to read data from the repos. Perhaps this isn't correct, but I'm not sure it's been fully tested yet.

Declaring this ticket complete, now that https://lsst-sandbox.ncsa.illinois.edu/pipetask-plot-navigator/dashboard_gen3 is live and working.