Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-584

New Conda environment for Science Pipelines

    XMLWordPrintable

Details

    • RFC
    • Status: Implemented
    • Resolution: Done
    • DM
    • None

    Description

      Now that v17.0 has been released, it seems the right moment to update the Conda environment.

      The following package versions will be updated:

      • python   3.7.2  (instead of 3.6.6)
      • astropy   3.1.2  (instead of 3.0.3)
      • numpy   1.16.2 (instead of 1.14.5)
      • matplotlib   3.0.3  (instead of 2.2.2)
      • scipy     1.2.1   (instead of 1.1.0)

      This update will also include the deprecated and pytables packages approved in RFC-557 and RFC-563.

      For a complete list of packages resolved, refer to:

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-linux-64.yml

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-osx-64.yml

      The proposed environment has been tested with lsst_ci, ci_hsc and lsst_distrib.

      Together with the update of the conda environment, lsstsw and newinstall are updated in order to use:

      • EUPS 2.1.5
      • miniconda 4.5.12

      Please note that the python version baselined in the Developer Guide is affected by this RFC.

       

      Attachments

        Issue Links

          Activity

            I have added a sentence to the RFC text.

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - I have added a sentence to the RFC text.
            ktl Kian-Tat Lim added a comment -

            I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about.

            I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env.  The latter is the implementation of policy, not the source of policy.

            I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?

            ktl Kian-Tat Lim added a comment - I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about. I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env .  The latter is the implementation of policy, not the source of policy. I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?
            ktl Kian-Tat Lim added a comment -

            P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"?

            And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.

            ktl Kian-Tat Lim added a comment - P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"? And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.
            tjenness Tim Jenness added a comment - - edited

            Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible.

            I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.

            tjenness Tim Jenness added a comment - - edited Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible. I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.

            The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

            People

              gcomoretto Gabriele Comoretto [X] (Inactive)
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Colin Slater, Fabio Hernandez, Gabriele Comoretto [X] (Inactive), John Parejko, John Swinbank, Kian-Tat Lim, Leanne Guy, Michelle Butler [X] (Inactive), Russell Owen, Scott Daniel, Tim Jenness, Wil O'Mullane
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.