Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:
-
Location:PR or chat room
Description
Currently the stack targets python version 2.7 exclusively. In the current timeline python 2.7 will no longer receive anything other than critical fixes and support for that will be ended in 2020. It will not be receiving any of the new features being added to python 3 (such as type hinting or matrix operators).
Python 3 is being adopted in the astronomy community and over the lifetime of the DM stack development it is assumed that python 3 will begin to dominate (there was even a recent discussion on the astropy mailing list where the idea of dropping support for 2.7 was actively discussed during the LTS cycle). We can not afford to defer the adoption of python 3 to the end of the development process. This RFC proposes that we start to modify the stack code such that it can run on python 3 and python 2.7. This is not a proposal to drop python 2.7 support.
Migrating to python 3 will be an incremental process rather than a big bang and will depend on external dependencies migrating. I advocate that we start the process of migrating before all dependencies are themselves ported. There is a gain in writing new code in a portable manner now rather than attempting to port it later.
If we accept the principle that striving for python 3 is a good goal then we can investigate using the future portability package (and associated futurize script) http://python-future.org/quickstart.html. We should also setup a single CI process to run with python3 that will initially break all the time but will also tell us the porting priorities.
Attachments
Issue Links
- is triggering
-
DM-6179 Support Python 3 migration
- Done
-
- Invalid
-
- Invalid
-
DM-5018 Modernize version check scripts in matplotlib and numpy packages
- Done
-
- Done
-
- Invalid
- relates to
-
- Invalid
-
- Implemented
Activity
Epics for test coverage include "top-down" starting with integration tests (that I think will be added as soon as the HSC merge gets into full swing and when DavidN starts) and "bottom-up" starting with better unit tests (that I have been advocating and will eventually start requiring; after we get the build/CI system to be "more standard", adding test coverage reports should be a relatively small task).
We are already supposed to be writing "more-compatible" code using from __future__ import.
Perhaps it will help if I propose a roadmap as Tim requested above. This is not an RFC, because there is nothing immediately actionable:
- Prerequisites before we do anything other than continue to write more-compatible code: Build/CI system in place and cleaned up, test coverage reports, integration tests, scons (if still the build tool) runnable under Python 3
- Create Python 3 build platform in CI system; run all code under Python 3 (expected to break).
- Do whatever is necessary to make application services middleware and process control middleware (porcupine legs) run under both Python 2 and Python 3 (e.g. run futurize); write new code to be compatible with both. This is a "big bang" because I want to avoid making developers switch gears back and forth during their daily work. I think a file-by-file or package-by-package transition, like the current transition to the "more-compatible" standard, is too slow. Old code doesn't get converted, and more importantly people's habits don't get converted, which is partly why Tim is finding code that doesn't adhere to the new standard.
- Do the same with afw (or whatever the porcupine body is at that time).
- Do the same with the remainder of the Science Pipelines code. Python 3 in CI goes full green. (These last two might have to be combined if there's a lot of overlap in developers.)
- When we find Python 3 features or libraries that substantially enhance our code or science that cannot be back-ported, we drop Python 2 compatibility.
I would like to reopen this for discussion at Bremerton next week. In particular we need to:
- Agree on the principal
- Decide how much test coverage is required before we can start moving to 2.7+3.4
- Agree that scons and eups will not be part of the conversion (it has been stated that eups must support python2.4).
- Enable a Python3 CI
This was discussed at LSST2015 in Bremerton. The decision was that we should move to python3 + 2.7 using future with the following requirements:
- Document resources on Python3 migration.
- Offer to give a talk on Python3 if there is demand.
- Provide example code migrations using DM code.
- Ensure that SIMs are aware of the changes.
- A python3 CI system will be created where scons will be called with python2.7.
An issue was raised as to whether the change to unicode literals in stack code would change the external API for python2.7 users. A naive investigation indicates that the lax support for unicode in python2.7 leads to this being a non-issue (Python2.7 assumes byte strings and unicode strings are interchangeable) although we should continue to be vigilant.
It was decided that this work can begin after the Summer15 release but packages with pending HSC merges will be deferred until later in the cycle. People will be allocated story points as part of Winter16 planning to migrate packages one at a time.
It was also decided that an RFC will also be issued proposing that server-side code and code written for internal maintenance (i.e. code that is not expected to be used by the community) can be written in native Python3 without worrying about Python2.7.
A summary of this decision was also tweeted:https://twitter.com/mjuric/status/634990661033287680
After discussion with Kian-Tat Lim I am marking this RFC as Implemented given that:
- Documentation now exists with a guide to porting.
- The developer guide has been updated.
- newinstall and lsstsw work.
- Jenkins supports Python 3.
- Packages at meas_ level and below now work.
- Remaining packages are now scheduled.
For the record, my opinion is that a big bang change years from now will consume more resources (and therefore be more difficult to get approved) than an evolutionary approach.