Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22030

Add panel + dependencies to shared stack

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      Add the dependencies for the QA dashboard to the shared stack. These are listed in this environment file here: https://github.com/Quansight/lsst_dashboard/blob/master/environment.yml

      Packages in that file for documentation (e.g., sphinx, nbsite, recommonmark, and sphinx_rtd_theme) are not required, and the only specific version requirement we are aware of is panel >= 0.6.3.

      Attachments

        Issue Links

          Activity

            No builds found.
            tmorton Tim Morton [X] (Inactive) created issue -

            We also need a more up-to-date version of pyarrow (>= 0.13.1, I believe).

            tmorton Tim Morton [X] (Inactive) added a comment - We also need a more up-to-date version of pyarrow (>= 0.13.1, I believe).

            Just to ping again here, I'm still iterating/updating things myself with the dashboard in order to have what is hopefully a useful product; stay tuned for the final dependencies.

            tmorton Tim Morton [X] (Inactive) added a comment - Just to ping again here, I'm still iterating/updating things myself with the dashboard in order to have what is hopefully a useful product; stay tuned for the final dependencies.

            swinbank Are you able to add the following packages to the shared stack? I'm not exactly sure they are all absolutely necessary, so let me know if there are problems with any of these and I can check, but this is what I currently have installed in my .local in order to run the dashboard:

            panel==0.8.0
            holoviews==1.12.7
            hvplot==0.5.2
            bokeh==1.4.0
            pyviz_comms==0.7.3
            fastparquet==0.3.2
            numba==0.46.0
            datashader==0.8.0
            pyct==0.4.6
            dask_jobqueue==0.7.0 [not strictly necessary for current dashboard, but will become useful soon]

            Some of these certainly come as dependencies of others, but I don't remember exactly, so this should be a complete list.

            tmorton Tim Morton [X] (Inactive) added a comment - swinbank Are you able to add the following packages to the shared stack? I'm not exactly sure they are all absolutely necessary, so let me know if there are problems with any of these and I can check, but this is what I currently have installed in my .local in order to run the dashboard: panel==0.8.0 holoviews==1.12.7 hvplot==0.5.2 bokeh==1.4.0 pyviz_comms==0.7.3 fastparquet==0.3.2 numba==0.46.0 datashader==0.8.0 pyct==0.4.6 dask_jobqueue==0.7.0 [not strictly necessary for current dashboard, but will become useful soon] Some of these certainly come as dependencies of others, but I don't remember exactly, so this should be a complete list.
            swinbank John Swinbank added a comment - - edited

            We need to build a new shared stack on DM-23349, which is currently blocked on DM-22817. Assuming the latter comes unstuck in the next day or two, I'll roll these into the new stack. Otherwise, I'll try to retrofit them to the old one.

            swinbank John Swinbank added a comment - - edited We need to build a new shared stack on DM-23349 , which is currently blocked on DM-22817 . Assuming the latter comes unstuck in the next day or two, I'll roll these into the new stack. Otherwise, I'll try to retrofit them to the old one.
            swinbank John Swinbank made changes -
            Field Original Value New Value
            Link This issue relates to DM-23349 [ DM-23349 ]
            swinbank John Swinbank made changes -
            Urgent? off

            If we could add geoviews and geopandas to the shared stack as well, we should be able to do the following in the dashboard without needing to request new packages:

            https://nbviewer.jupyter.org/github/Quansight/lsst_dashboard/blob/master/notebooks/tract-viz.ipynb

            tmorton Tim Morton [X] (Inactive) added a comment - If we could add geoviews and geopandas to the shared stack as well, we should be able to do the following in the dashboard without needing to request new packages: https://nbviewer.jupyter.org/github/Quansight/lsst_dashboard/blob/master/notebooks/tract-viz.ipynb

            swinbank regarding the above recent request for geoviews/geopandas, the geo-package dependencies are difficult to navigate. Accumulated wisdom from the quansight team suggests that conda-installing from conda-forge should do the trick, but not to try pip-installing. As discussed with krughoff, for the LSP, taking this approach would mean a shift from a pip-centric to conda-centric docker setup, which would take some significant effort, so hopefully it will be easier on this stack?

            tmorton Tim Morton [X] (Inactive) added a comment - swinbank regarding the above recent request for geoviews/geopandas, the geo-package dependencies are difficult to navigate. Accumulated wisdom from the quansight team suggests that conda-installing from conda-forge should do the trick, but not to try pip-installing. As discussed with krughoff , for the LSP, taking this approach would mean a shift from a pip-centric to conda-centric docker setup, which would take some significant effort, so hopefully it will be easier on this stack?

            I have dask-jobqueue==0.7.0 (not dask_jobqueue). I take it that's the right thing.

            swinbank John Swinbank added a comment - I have dask-jobqueue==0.7.0 (not dask_jobqueue ). I take it that's the right thing.
            swinbank John Swinbank added a comment - - edited

            Looks like I can get everything into the new stack except geopandas. Geoviews went in fine, but conda install -c conda-forge --no-update-deps geopandas thinks for a very long time then gives me an UnsatisfiableError. (We need the --no-update-deps because we need to keep the base scipipe_conda_env unchanged.)

            How acceptable is that to you tmorton?

            swinbank John Swinbank added a comment - - edited Looks like I can get everything into the new stack except geopandas. Geoviews went in fine, but conda install -c conda-forge --no-update-deps geopandas thinks for a very long time then gives me an UnsatisfiableError . (We need the --no-update-deps because we need to keep the base scipipe_conda_env unchanged.) How acceptable is that to you tmorton ?
            swinbank John Swinbank made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]

            Stack containing the latest weekly and all the requested deps except geopandas (as above) now available at /software/lsstsw/stack/20200220. I'll make this the default shared stack later today. Tim, please let me know if there's anything else you need.

            swinbank John Swinbank added a comment - Stack containing the latest weekly and all the requested deps except geopandas (as above) now available at /software/lsstsw/stack/20200220 . I'll make this the default shared stack later today. Tim, please let me know if there's anything else you need.
            swinbank John Swinbank made changes -
            Reviewers Tim Morton [ tmorton ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tmorton Tim Morton [X] (Inactive) added a comment - - edited

            Thanks John, sorry I didn't get around to checking this last week. Looking at it now; is there some reason I shouldn't be able to use pip in this stack?

            (lsst-scipipe) [tmorton@lsst-dev01 ~]$ pip install geopandas --user
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/bin/pip", line 7, in <module>
                from pip._internal.cli.main import main
            ModuleNotFoundError: No module named 'pip._internal.cli.main'
            

            tmorton Tim Morton [X] (Inactive) added a comment - - edited Thanks John, sorry I didn't get around to checking this last week. Looking at it now; is there some reason I shouldn't be able to use pip in this stack? (lsst-scipipe) [tmorton@lsst-dev01 ~]$ pip install geopandas --user Traceback (most recent call last): File "/software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/bin/pip", line 7, in <module> from pip._internal.cli.main import main ModuleNotFoundError: No module named 'pip._internal.cli.main'

            Unhelpfully, perhaps, I can say “works for me!”:

            Enabling rh-git29 devtoolset-8.
            [swinbank@lsst-dev01 ~]$ . /software/lsstsw/stack_20200220/loadLSST.bash
            [swinbank@lsst-dev01 ~]$ pip install geopandas --user
            Collecting geopandas
              Using cached geopandas-0.7.0-py2.py3-none-any.whl (928 kB)
            Requirement already satisfied: shapely in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (1.6.4.post2)
            Requirement already satisfied: pyproj>=2.2.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (2.3.1)
            Requirement already satisfied: pandas>=0.23.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (1.0.1)
            Collecting fiona
              Using cached Fiona-1.8.13.post1-cp37-cp37m-manylinux1_x86_64.whl (14.7 MB)
            Requirement already satisfied: python-dateutil>=2.6.1 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (2.8.1)
            Requirement already satisfied: numpy>=1.13.3 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (1.18.1)
            Requirement already satisfied: pytz>=2017.2 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (2019.3)
            Requirement already satisfied: click<8,>=4.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (7.0)
            Requirement already satisfied: attrs>=17 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (19.3.0)
            Collecting click-plugins>=1.0
              Using cached click_plugins-1.1.1-py2.py3-none-any.whl (7.5 kB)
            Requirement already satisfied: six>=1.7 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (1.14.0)
            Collecting munch
              Using cached munch-2.5.0-py2.py3-none-any.whl (10 kB)
            Collecting cligj>=0.5
              Using cached cligj-0.5.0-py3-none-any.whl (5.7 kB)
            Installing collected packages: click-plugins, munch, cligj, fiona, geopandas
            Successfully installed click-plugins-1.1.1 cligj-0.5.0 fiona-1.8.13.post1 geopandas-0.7.0 munch-2.5.0
            

            I do note https://github.com/pypa/pip/issues/5599, which makes me wonder if this is due to local modifications to your environment. Do you have your own version of pip installed somewhere? Can you try e.g. getting rid of ~/.local to see if that helps?

            swinbank John Swinbank added a comment - Unhelpfully, perhaps, I can say “works for me!”: Enabling rh-git29 devtoolset-8. [swinbank@lsst-dev01 ~]$ . /software/lsstsw/stack_20200220/loadLSST.bash [swinbank@lsst-dev01 ~]$ pip install geopandas --user Collecting geopandas Using cached geopandas-0.7.0-py2.py3-none-any.whl (928 kB) Requirement already satisfied: shapely in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (1.6.4.post2) Requirement already satisfied: pyproj>=2.2.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (2.3.1) Requirement already satisfied: pandas>=0.23.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from geopandas) (1.0.1) Collecting fiona Using cached Fiona-1.8.13.post1-cp37-cp37m-manylinux1_x86_64.whl (14.7 MB) Requirement already satisfied: python-dateutil>=2.6.1 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (2.8.1) Requirement already satisfied: numpy>=1.13.3 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (1.18.1) Requirement already satisfied: pytz>=2017.2 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from pandas>=0.23.0->geopandas) (2019.3) Requirement already satisfied: click<8,>=4.0 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (7.0) Requirement already satisfied: attrs>=17 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (19.3.0) Collecting click-plugins>=1.0 Using cached click_plugins-1.1.1-py2.py3-none-any.whl (7.5 kB) Requirement already satisfied: six>=1.7 in /software/lsstsw/stack_20200220/python/miniconda3-4.7.12/envs/lsst-scipipe/lib/python3.7/site-packages (from fiona->geopandas) (1.14.0) Collecting munch Using cached munch-2.5.0-py2.py3-none-any.whl (10 kB) Collecting cligj>=0.5 Using cached cligj-0.5.0-py3-none-any.whl (5.7 kB) Installing collected packages: click-plugins, munch, cligj, fiona, geopandas Successfully installed click-plugins-1.1.1 cligj-0.5.0 fiona-1.8.13.post1 geopandas-0.7.0 munch-2.5.0 I do note https://github.com/pypa/pip/issues/5599 , which makes me wonder if this is due to local modifications to your environment. Do you have your own version of pip installed somewhere? Can you try e.g. getting rid of ~/.local to see if that helps?

            Good suggestion; thanks! That removes that issue. Will continue testing now.

            tmorton Tim Morton [X] (Inactive) added a comment - Good suggestion; thanks! That removes that issue. Will continue testing now.

            So I don't get the same segfault I was getting on the LSP, so that's promising, but it's not actually visualizing the geoviews plots (though holoviews is OK) on the jupyter nb I was testing on.

            Aside from that, however, it looks like the QA Dashboard all works great with the default shared stack, which is great.

            tmorton Tim Morton [X] (Inactive) added a comment - So I don't get the same segfault I was getting on the LSP, so that's promising, but it's not actually visualizing the geoviews plots (though holoviews is OK) on the jupyter nb I was testing on. Aside from that, however, it looks like the QA Dashboard all works great with the default shared stack, which is great.
            swinbank John Swinbank added a comment - - edited

            I'm not sure if I should parse that as this ticket being a success, a failure, or just that you're still working on the review!

            swinbank John Swinbank added a comment - - edited I'm not sure if I should parse that as this ticket being a success, a failure, or just that you're still working on the review!

            I'd say about 99%, and 1% I don't know what's going on, but we're planning to work around geoviews anyway, so that 1% is pretty low priority to figure out what's going on.

            tmorton Tim Morton [X] (Inactive) added a comment - I'd say about 99%, and 1% I don't know what's going on, but we're planning to work around geoviews anyway, so that 1% is pretty low priority to figure out what's going on.

            I think I'm going to take that as a review, and invite you to file more tickets if you need further support!

            swinbank John Swinbank added a comment - I think I'm going to take that as a review, and invite you to file more tickets if you need further support!
            swinbank John Swinbank made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            swinbank John Swinbank made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Team Data Release Production [ 10301 ] Alert Production [ 10300 ]
            swinbank John Swinbank made changes -
            Epic Link DM-22484 [ 427311 ]

            People

              swinbank John Swinbank
              tmorton Tim Morton [X] (Inactive)
              Tim Morton [X] (Inactive)
              John Swinbank, Tim Morton [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.