Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-584

New Conda environment for Science Pipelines

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Now that v17.0 has been released, it seems the right moment to update the Conda environment.

      The following package versions will be updated:

      • python   3.7.2  (instead of 3.6.6)
      • astropy   3.1.2  (instead of 3.0.3)
      • numpy   1.16.2 (instead of 1.14.5)
      • matplotlib   3.0.3  (instead of 2.2.2)
      • scipy     1.2.1   (instead of 1.1.0)

      This update will also include the deprecated and pytables packages approved in RFC-557 and RFC-563.

      For a complete list of packages resolved, refer to:

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-linux-64.yml

      https://github.com/lsst/scipipe_conda_env/blob/tickets/DM-17684/etc/conda3_packages-osx-64.yml

      The proposed environment has been tested with lsst_ci, ci_hsc and lsst_distrib.

      Together with the update of the conda environment, lsstsw and newinstall are updated in order to use:

      • EUPS 2.1.5
      • miniconda 4.5.12

      Please note that the python version baselined in the Developer Guide is affected by this RFC.

       

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            +1 on updating to Python 3.7.2. The other updates also sound like a great idea (assuming they don't break anything).

            Show
            rowen Russell Owen added a comment - +1 on updating to Python 3.7.2. The other updates also sound like a great idea (assuming they don't break anything).
            Hide
            tjenness Tim Jenness added a comment -

            I am in favor of these updates.

            Show
            tjenness Tim Jenness added a comment - I am in favor of these updates.
            Hide
            swinbank John Swinbank added a comment -

            Just to be clear, is the proposal just to update the Conda environment (which is what the RFC says), or also to update the documented baseline Python version?

            Show
            swinbank John Swinbank added a comment - Just to be clear, is the proposal just to update the Conda environment (which is what the RFC says), or also to update the documented baseline Python version ?
            Hide
            tjenness Tim Jenness added a comment -

            We are using the term "Conda environment" to mean the conda environment in the full sense of "conda packages defining the environment in which to run science pipelines." That includes python3.7.

            Show
            tjenness Tim Jenness added a comment - We are using the term "Conda environment" to mean the conda environment in the full sense of "conda packages defining the environment in which to run science pipelines." That includes python3.7.
            Hide
            swinbank John Swinbank added a comment -

            I think what you're saying is that you want to change the Developer Guide to say that the baseline is defined by the Conda environment. Is that right?

            Show
            swinbank John Swinbank added a comment - I think  what you're saying is that you want to change the Developer Guide to say that the baseline is defined by the Conda environment. Is that right?
            Hide
            tjenness Tim Jenness added a comment -

            John Swinbank I believe that would be the best approach since the conda environment defines everything anyhow.

            Show
            tjenness Tim Jenness added a comment - John Swinbank I believe that would be the best approach since the conda environment defines everything anyhow.
            Hide
            swinbank John Swinbank added a comment -

            I'm happy with that, but I think we need to make it really explicit on this RFC: this is not just about updating conda.

            Show
            swinbank John Swinbank added a comment - I'm happy with that, but I think we need to make it really explicit on this RFC: this is not just about updating conda.
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            I support the idea of upgrading conda and the packages mentioned in this RFC. That will allow us to extend lsst_distrib with additional packages, also installable via conda, which may depend on recent version of some "foundation" packages. I would even propose to consider upgrading conda and friends at regular intervals, say once per quarter, if possible.

            Show
            FabioHernandez Fabio Hernandez added a comment - I support the idea of upgrading conda  and the packages mentioned in this RFC. That will allow us to extend lsst_distrib with additional packages, also installable via conda , which may depend on recent version of some "foundation" packages. I would even propose to consider upgrading conda and friends at regular intervals, say once per quarter, if possible.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            If this RFC is approved, the developer guide needs to be updated also. However, this is a different topic and can be discussed on a triggered issue.

            I think Kian-Tat Lim may have an opinion on how to do it.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - If this RFC is approved, the developer guide needs to be updated also. However, this is a different topic and can be discussed on a triggered issue. I think Kian-Tat Lim may have an opinion on how to do it.
            Hide
            swinbank John Swinbank added a comment -

            To be clear, “the Developer Guide needs to be updated” is tangential to the point here. The real issue is that this RFC proposes a change to the documented Python baseline, not just a bump to the Conda environment. That's not explicit from the RFC as written.

            Show
            swinbank John Swinbank added a comment - To be clear, “the Developer Guide needs to be updated” is tangential to the point here. The real issue is that this RFC proposes a change to the documented Python baseline, not just a bump to the Conda environment. That's not explicit from the RFC as written.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            I have added a sentence to the RFC text.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - I have added a sentence to the RFC text.
            Hide
            ktl Kian-Tat Lim added a comment -

            I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about.

            I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env.  The latter is the implementation of policy, not the source of policy.

            I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?

            Show
            ktl Kian-Tat Lim added a comment - I agree that we should update the Developer Guide to state that we are using Python 3.7, and enabling use of all of its features.  As I understand it, there are no backward compatibility issues to worry about. I don't think that we should have the Developer Guide state that the Python version is whatever is listed in scipipe_conda_env .  The latter is the implementation of policy, not the source of policy. I also don't believe that we are updating the version of miniconda, which is still at 4.5.4 in newinstall.sh and is not mentioned in this RFC.  Is this correct?
            Hide
            ktl Kian-Tat Lim added a comment -

            P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"?

            And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.

            Show
            ktl Kian-Tat Lim added a comment - P.S.  Can someone publicly write up or point to a "summary of Python 3.7 features of note for LSST developers"? And I'll note here that there may be indirect impacts on T&S and other CSC developers from our upgrading Python, but I think these are acceptable or can be mitigated.
            Hide
            tjenness Tim Jenness added a comment - - edited

            Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible.

            I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.

            Show
            tjenness Tim Jenness added a comment - - edited Telescope and site are asking me to go to 3.7. It's also the case that 3.7 has been tested for months and T&S building their own docker container using 3.7 is entirely plausible. I believe that we are updating miniconda version to 4.5.12. (the most recent). There is a bug fixed in conda 4.6 that we would like (conda env export is getting confused by pytables and lists it as a pip package and conda package in 4.5.x) but until a native miniconda is released using it it's a bit complicated to do the update in newinstall/deploy logic.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - The miniconda we are updating to is 4..5.12. I make it explicit in the description. Sorry for missing it.

              People

              Assignee:
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Reporter:
              gcomoretto Gabriele Comoretto [X] (Inactive)
              Watchers:
              Colin Slater, Fabio Hernandez, Gabriele Comoretto [X] (Inactive), John Parejko, John Swinbank, Kian-Tat Lim, Leanne Guy, Michelle Butler, Russell Owen, Scott Daniel, Tim Jenness, Wil O'Mullane
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  CI Builds

                  No builds found.