Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-382

Update minimum Python3 version to 3.6 after v14 is released

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:

      Description

      We currently mandate that DM software must work with python 3.5 and above and that science pipelines must also support python 2.7. After v14 is released I would like to shift the DM baseline for python3 to be v3.6.

      My motivation for doing this is so that we can agree on a minimum version when integration testing ramps up in 2018 and there is little point running tests on a version of python that we have no intention of using in production (whereas aux telescope will be arriving next year). Given that science pipelines still has to support py2.7, this won't have any visible effect on developers other than the python package insisting that 3.6 is available. Many developers are already running 3.6 but Jenkins is not testing it.

        Attachments

          Issue Links

            Activity

            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Link This issue relates to RFC-379 [ RFC-379 ]
            mjuric Mario Juric made changes -
            Labels dm-sst
            Hide
            mjuric Mario Juric added a comment -

            What is the readily-available version of Py3 on our reference platform (is it still CentOS 7?). Or would the version that comes with Anaconda be more relevant?

            Show
            mjuric Mario Juric added a comment - What is the readily-available version of Py3 on our reference platform (is it still CentOS 7?). Or would the version that comes with Anaconda be more relevant?
            Hide
            tjenness Tim Jenness added a comment -

            3.6 is the standard miniconda version. We currently force an install of an older python. The CentOS python version is not relevant.

            Show
            tjenness Tim Jenness added a comment - 3.6 is the standard miniconda version. We currently force an install of an older python. The CentOS python version is not relevant.
            Hide
            mjuric Mario Juric added a comment -

            I'd naively expect we'd try to use as many tools from the reference platform as we can (esp. compilers & interpreters). If CentOS7+Anaconda vX.Y have de-facto become our reference platform, should we just say so?

            Re the minimum supported version of Python 3: my primary concern is that we don't have to change it often (especially during commissioning or early ops). If the switch is made now to 3.6, will it remain so until ~2023?

            Show
            mjuric Mario Juric added a comment - I'd naively expect we'd try to use as many tools from the reference platform as we can (esp. compilers & interpreters). If CentOS7+Anaconda vX.Y have de-facto become our reference platform, should we just say so? Re the minimum supported version of Python 3: my primary concern is that we don't have to change it often (especially during commissioning or early ops). If the switch is made now to 3.6, will it remain so until ~2023?
            Hide
            tjenness Tim Jenness added a comment -

            Re: Anaconda. I'm not suggesting we make that the standard. Macs don't even come with a python3 so you have to install your own somehow and if you use homebrew or anaconda you will be getting 3.6. To get a python3 on CentOS7 I think you need to use a new yum repo.

            I don't see any gain in ensuring things work with older python releases. Supporting different operating systems and compiler vendors teaches us something about our code quality. Supporting an older python release when we don't need to does not contribute to code quality and uses more resources.

            We can't use python 3.6 in 2023 because it won't be supported. Python has 5 year support plans. I imagine we would need to be using python 3.7 in early-2019 (it's due to come out the middle of 2018). Kian-Tat Lim also made the point on Slack that doing one python uprev during 2018-2022 will be useful for testing the process.

            Show
            tjenness Tim Jenness added a comment - Re: Anaconda. I'm not suggesting we make that the standard. Macs don't even come with a python3 so you have to install your own somehow and if you use homebrew or anaconda you will be getting 3.6. To get a python3 on CentOS7 I think you need to use a new yum repo. I don't see any gain in ensuring things work with older python releases. Supporting different operating systems and compiler vendors teaches us something about our code quality. Supporting an older python release when we don't need to does not contribute to code quality and uses more resources. We can't use python 3.6 in 2023 because it won't be supported. Python has 5 year support plans. I imagine we would need to be using python 3.7 in early-2019 (it's due to come out the middle of 2018). Kian-Tat Lim also made the point on Slack that doing one python uprev during 2018-2022 will be useful for testing the process.
            Hide
            mjuric Mario Juric added a comment - - edited

            I see your point, though I can't say I fully agree. I'm not thinking of this as "having to support an older release", but more as "not wanting to be on the bleeding edge release" (or making changes at critical times). Personally prefer to let others discover their new bugs & focus on my own .

            But this may be different & I'll defer to your collective wisdom on what's best.

            Show
            mjuric Mario Juric added a comment - - edited I see your point, though I can't say I fully agree. I'm not thinking of this as "having to support an older release", but more as "not wanting to be on the bleeding edge release" (or making changes at critical times). Personally prefer to let others discover their new bugs & focus on my own . But this may be different & I'll defer to your collective wisdom on what's best.
            Hide
            tjenness Tim Jenness added a comment -

            I'm wanting to go to 3.6 now specifically so that we can do integration and testing in early 2018 without having to jump on a new version then. Many of us are already using 3.6. Given that we do support 2.7 still, it's possible to switch baseline anaconda to 3.6 but still run periodic tests on Jenkins with 3.5 (which I think was the bit that worried John Swinbank) but we haven't got the infrastructure in place in lsstsw to support two different versions of python3 package pinning and I'm not sure it gains us much.

            Show
            tjenness Tim Jenness added a comment - I'm wanting to go to 3.6 now specifically so that we can do integration and testing in early 2018 without having to jump on a new version then. Many of us are already using 3.6. Given that we do support 2.7 still, it's possible to switch baseline anaconda to 3.6 but still run periodic tests on Jenkins with 3.5 (which I think was the bit that worried John Swinbank ) but we haven't got the infrastructure in place in lsstsw to support two different versions of python3 package pinning and I'm not sure it gains us much.
            Hide
            swinbank John Swinbank added a comment -

            Given that we do support 2.7 still, it's possible to switch baseline anaconda to 3.6 but still run periodic tests on Jenkins with 3.5 (which I think was the bit that worried John Swinbank)

            I'm not sure that John Swinbank is particularly worried by this proposal. And I'm not sure he understands the above, either — if we switch to 3.6, I don't think we need to worry about testing on versions older than that which we have declared support for.

            We must be rigorous about testing the versions we do support, though — once we say we support 3.6 (or certain versions of Matplotlib, or whatever), then we must test against those versions until we decide to drop support for them: no letting pinned versions creep upwards without changing the documented requirements.

            Show
            swinbank John Swinbank added a comment - Given that we do support 2.7 still, it's possible to switch baseline anaconda to 3.6 but still run periodic tests on Jenkins with 3.5 (which I think was the bit that worried John Swinbank) I'm not sure that John Swinbank is particularly worried by this proposal. And I'm not sure he understands the above, either — if we switch to 3.6, I don't think we need to worry about testing on versions older than that which we have declared support for. We must be rigorous about testing the versions we do support, though — once we say we support 3.6 (or certain versions of Matplotlib, or whatever), then we must test against those versions until we decide to drop support for them: no letting pinned versions creep upwards without changing the documented requirements.
            Hide
            tjenness Tim Jenness added a comment -

            Yes. I'm fine with that, I would like us to do a weekly "build against all the latest versions" job in addition to pinning to the declared versions. That way we get a heads up for future breakage without waiting for the community to tell us and we can plan the work to fix things up.

            Show
            tjenness Tim Jenness added a comment - Yes. I'm fine with that, I would like us to do a weekly "build against all the latest versions" job in addition to pinning to the declared versions. That way we get a heads up for future breakage without waiting for the community to tell us and we can plan the work to fix things up.
            Hide
            tjenness Tim Jenness added a comment -

            No objections. I'll update this when updating pinned versions for RFC-379.

            Show
            tjenness Tim Jenness added a comment - No objections. I'll update this when updating pinned versions for RFC-379 .
            tjenness Tim Jenness made changes -
            Status Proposed [ 10805 ] Adopted [ 10806 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-11727 [ DM-11727 ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                tjenness Tim Jenness
                Watchers:
                John Swinbank, Mario Juric, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel