Tim Morton [X] (Inactive) added a comment - 31/Oct/19 7:35 PM We also need a more up-to-date version of pyarrow (>= 0.13.1, I believe).

Tim Morton [X] (Inactive) added a comment - 03/Dec/19 7:49 PM Just to ping again here, I'm still iterating/updating things myself with the dashboard in order to have what is hopefully a useful product; stay tuned for the final dependencies.

Tim Morton [X] (Inactive) added a comment - 12/Feb/20 7:46 PM John Swinbank Are you able to add the following packages to the shared stack? I'm not exactly sure they are all absolutely necessary, so let me know if there are problems with any of these and I can check, but this is what I currently have installed in my .local in order to run the dashboard: panel==0.8.0 holoviews==1.12.7 hvplot==0.5.2 bokeh==1.4.0 pyviz_comms==0.7.3 fastparquet==0.3.2 numba==0.46.0 datashader==0.8.0 pyct==0.4.6 dask_jobqueue==0.7.0 [not strictly necessary for current dashboard, but will become useful soon] Some of these certainly come as dependencies of others, but I don't remember exactly, so this should be a complete list.

John Swinbank added a comment - 12/Feb/20 11:51 PM - edited We need to build a new shared stack on DM-23349 , which is currently blocked on DM-22817 . Assuming the latter comes unstuck in the next day or two, I'll roll these into the new stack. Otherwise, I'll try to retrofit them to the old one.

Tim Morton [X] (Inactive) added a comment - 18/Feb/20 11:37 PM If we could add geoviews and geopandas to the shared stack as well, we should be able to do the following in the dashboard without needing to request new packages: https://nbviewer.jupyter.org/github/Quansight/lsst_dashboard/blob/master/notebooks/tract-viz.ipynb

Tim Morton [X] (Inactive) added a comment - 19/Feb/20 9:16 PM John Swinbank regarding the above recent request for geoviews/geopandas, the geo-package dependencies are difficult to navigate. Accumulated wisdom from the quansight team suggests that conda-installing from conda-forge should do the trick, but not to try pip-installing. As discussed with Simon Krughoff , for the LSP, taking this approach would mean a shift from a pip-centric to conda-centric docker setup, which would take some significant effort, so hopefully it will be easier on this stack?

John Swinbank added a comment - 21/Feb/20 12:28 AM I have dask-jobqueue==0.7.0 (not dask_jobqueue ). I take it that's the right thing.

John Swinbank added a comment - 21/Feb/20 1:27 AM - edited Looks like I can get everything into the new stack except geopandas. Geoviews went in fine, but conda install -c conda-forge --no-update-deps geopandas thinks for a very long time then gives me an UnsatisfiableError . (We need the --no-update-deps because we need to keep the base scipipe_conda_env unchanged.) How acceptable is that to you Tim Morton [X] ?

John Swinbank added a comment - 21/Feb/20 7:52 PM Stack containing the latest weekly and all the requested deps except geopandas (as above) now available at /software/lsstsw/stack/20200220 . I'll make this the default shared stack later today. Tim, please let me know if there's anything else you need.

Tim Morton [X] (Inactive) added a comment - 24/Feb/20 6:49 PM - edited Thanks John, sorry I didn't get around to checking this last week. Looking at it now; is there some reason I shouldn't be able to use pip in this stack? (lsst-scipipe) [tmorton@lsst-dev01 ~]$ pip install geopandas --user Traceback (most recent call last): File "/software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/bin/pip", line 7, in <module> from pip._internal.cli.main import main ModuleNotFoundError: No module named 'pip._internal.cli.main'

John Swinbank added a comment - 24/Feb/20 7:04 PM Unhelpfully, perhaps, I can say “works for me!”: Enabling rh-git29 devtoolset-8. [swinbank@lsst-dev01 ~]$ . /software/lsstsw/stack_20200220/loadLSST.bash [swinbank@lsst-dev01 ~]$ pip install geopandas --user Collecting geopandas Using cached geopandas-0.7.0-py2.py3-none-any.whl (928 kB) Requirement already satisfied: shapely in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (1.6.4.post2) Requirement already satisfied: pyproj>=2.2.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (2.3.1) Requirement already satisfied: pandas>=0.23.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (1.0.1) Collecting fiona Using cached Fiona-1.8.13.post1-cp37-cp37m-manylinux1_x86_64.whl (14.7 MB) Requirement already satisfied: python-dateutil>=2.6.1 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (2.8.1) Requirement already satisfied: numpy>=1.13.3 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (1.18.1) Requirement already satisfied: pytz>=2017.2 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (2019.3) Requirement already satisfied: click<8,>=4.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (7.0) Requirement already satisfied: attrs>=17 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (19.3.0) Collecting click-plugins>=1.0 Using cached click_plugins-1.1.1-py2.py3-none-any.whl (7.5 kB) Requirement already satisfied: six>=1.7 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (1.14.0) Collecting munch Using cached munch-2.5.0-py2.py3-none-any.whl (10 kB) Collecting cligj>=0.5 Using cached cligj-0.5.0-py3-none-any.whl (5.7 kB) Installing collected packages: click-plugins, munch, cligj, fiona, geopandas Successfully installed click-plugins-1.1.1 cligj-0.5.0 fiona-1.8.13.post1 geopandas-0.7.0 munch-2.5.0 I do note https://github.com/pypa/pip/issues/5599 , which makes me wonder if this is due to local modifications to your environment. Do you have your own version of pip installed somewhere? Can you try e.g. getting rid of ~/.local to see if that helps?

Tim Morton [X] (Inactive) added a comment - 24/Feb/20 7:17 PM Good suggestion; thanks! That removes that issue. Will continue testing now.

Tim Morton [X] (Inactive) added a comment - 24/Feb/20 10:10 PM So I don't get the same segfault I was getting on the LSP, so that's promising, but it's not actually visualizing the geoviews plots (though holoviews is OK) on the jupyter nb I was testing on. Aside from that, however, it looks like the QA Dashboard all works great with the default shared stack, which is great.

John Swinbank added a comment - 24/Feb/20 11:25 PM - edited I'm not sure if I should parse that as this ticket being a success, a failure, or just that you're still working on the review!

Tim Morton [X] (Inactive) added a comment - 25/Feb/20 4:38 AM I'd say about 99%, and 1% I don't know what's going on, but we're planning to work around geoviews anyway, so that 1% is pretty low priority to figure out what's going on.

John Swinbank added a comment - 25/Feb/20 5:44 PM I think I'm going to take that as a review, and invite you to file more tickets if you need further support!