Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-584

New Conda environment for Science Pipelines

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Now that v17.0 has been released, it seems the right moment to update the Conda environment.

      The following package versions will be updated:

      • python   3.7.2  (instead of 3.6.6)
      • astropy   3.1.2  (instead of 3.0.3)
      • numpy   1.16.2 (instead of 1.14.5)
      • matplotlib   3.0.3  (instead of 2.2.2)
      • scipy     1.2.1   (instead of 1.1.0)

      This update will also include the deprecated and pytables packages approved in RFC-557 and RFC-563.

      For a complete list of packages resolved, refer to:

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-linux-64.yml

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-osx-64.yml

      The proposed environment has been tested with lsst_ci, ci_hsc and lsst_distrib.

      Together with the update of the conda environment, lsstsw and newinstall are updated in order to use:

      • EUPS 2.1.5
      • miniconda 4.5.12

      Please note that the python version baselined in the Developer Guide is affected by this RFC.

       

        Attachments

          Issue Links

            Activity

            Hide
            gcomoretto Gabriele Comoretto added a comment -

            I have added a sentence to the RFC text.

            Show
            gcomoretto Gabriele Comoretto added a comment - I have added a sentence to the RFC text.
            Hide
            ktl Kian-Tat Lim added a comment -

            I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about.

            I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env.  The latter is the implementation of policy, not the source of policy.

            I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?

            Show
            ktl Kian-Tat Lim added a comment - I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about. I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env .  The latter is the implementation of policy, not the source of policy. I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?
            Hide
            ktl Kian-Tat Lim added a comment -

            P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"?

            And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.

            Show
            ktl Kian-Tat Lim added a comment - P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"? And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.
            Hide
            tjenness Tim Jenness added a comment - - edited

            Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible.

            I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.

            Show
            tjenness Tim Jenness added a comment - - edited Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible. I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.
            Hide
            gcomoretto Gabriele Comoretto added a comment -

            The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

            Show
            gcomoretto Gabriele Comoretto added a comment - The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

              People

              • Assignee:
                gcomoretto Gabriele Comoretto
                Reporter:
                gcomoretto Gabriele Comoretto
                Watchers:
                Colin Slater, Fabio Hernandez, Gabriele Comoretto, John Parejko, John Swinbank, Kian-Tat Lim, Leanne Guy, Michelle Butler, Russell Owen, Scott Daniel, Tim Jenness, Wil O'Mullane
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel