Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-584

New Conda environment for Science Pipelines

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Now that v17.0 has been released, it seems the right moment to update the Conda environment.

      The following package versions will be updated:

      • python   3.7.2  (instead of 3.6.6)
      • astropy   3.1.2  (instead of 3.0.3)
      • numpy   1.16.2 (instead of 1.14.5)
      • matplotlib   3.0.3  (instead of 2.2.2)
      • scipy     1.2.1   (instead of 1.1.0)

      This update will also include the deprecated and pytables packages approved in RFC-557 and RFC-563.

      For a complete list of packages resolved, refer to:

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-linux-64.yml

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-osx-64.yml

      The proposed environment has been tested with lsst_ci, ci_hsc and lsst_distrib.

      Together with the update of the conda environment, lsstsw and newinstall are updated in order to use:

      • EUPS 2.1.5
      • miniconda 4.5.12

      Please note that the python version baselined in the Developer Guide is affected by this RFC.

       

        Attachments

          Issue Links

          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17684 Enable use of "deprecated" package in new conda environment

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18672 Update shared stack to new conda environment

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18691 Update documented python baselined version

          • Done
          relates to

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-379 Update minimum versions of numpy, astropy and matplotlib after v14.0 is released

          • Implemented

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-557 Deprecation procedure for methods and classes

          • Implemented

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-563 Adding python package tables

          • Implemented

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-664 Update conda base environment

          • Implemented

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-725 Update baseline python to version 3.8

          • Implemented

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22817 Update conda base environment

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17684 Enable use of "deprecated" package in new conda environment

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17721 Add package "tables" to scipipe_conda_env

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-18398 Update base EUPS and conda versions

          • Done
          mentioned in

          Page Loading...

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            10 older comments
            Hide
            Permalink
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            I have added a sentence to the RFC text.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - I have added a sentence to the RFC text.
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about.

            I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env.  The latter is the implementation of policy, not the source of policy.

            I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?

            Show
            ktl Kian-Tat Lim added a comment - I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about. I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env .  The latter is the implementation of policy, not the source of policy. I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"?

            And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.

            Show
            ktl Kian-Tat Lim added a comment - P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"? And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.
            Hide
            Permalink
            tjenness Tim Jenness added a comment - - edited

            Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible.

            I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.

            Show
            tjenness Tim Jenness added a comment - - edited Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible. I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.
            Hide
            Permalink
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

              People

              Assignee:
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Reporter:
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Watchers:
              Colin Slater, Fabio Hernandez, Gabriele Comoretto [X] (Inactive), John Parejko, John Swinbank, Kian-Tat Lim, Leanne Guy, Michelle Butler [X] (Inactive), Russell Owen, Scott Daniel, Tim Jenness, Wil O'Mullane
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.