Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-342

Use Python 3 for services and data challenges

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      I propose that during the F17 cycle we should switch our default Python from Python2.7 to Python3 for all services, integration tests, end to end processing tests and data challenges. This includes (but is not limited to):

      • DAX services and the PDAC (including Qserv admin, Firefly python integration),
      • Shared pipelines stack at NCSA.
      • HSC reprocessing.
      • Workflow testing at NCSA
      • Hosted JupyterLab notebooks

      All this code should be working on Python 3 already. Python 3 should be the default python that we use for testing and verification at NCSA. We should not be trying to switch major python versions during commissioning.

      I understand that there may be issues currently with the OCS components as I believe the SAL has not yet been ported to Python 3 (but it is in the plan). I will talk to Michael Reuter and Dave Mills about this but we should aim to switch our OCS python code to Python 3 as soon as is practical.

      I propose we use Python 3.6 (on the basis that there is no point starting on a version that is already old) and update lsstsw/newinstall appropriately.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            I'd like to see the ci_hsc problem addressed on the implementation ticket(s) for this (either by upgrading SCons or rewriting the scripts to only use our modules via a separate process, which would probably be better), but I'm otherwise in favor of this. Supporting two Python versions is causing a lot of pain, and while I don't think we can afford to drop Python 2 yet, that day is coming.

            Show
            jbosch Jim Bosch added a comment - I'd like to see the ci_hsc problem addressed on the implementation ticket(s) for this (either by upgrading SCons or rewriting the scripts to only use our modules via a separate process, which would probably be better), but I'm otherwise in favor of this. Supporting two Python versions is causing a lot of pain, and while I don't think we can afford to drop Python 2 yet, that day is coming.
            Hide
            tjenness Tim Jenness added a comment -

            I don't want this RFC to be seen as a starting point for dropping Python 2 everywhere. If we all agree that in operations we are going to be using Python 3 then it makes sense that we use Python 3 for integration and test activities. The earlier we do that for our internal testing the easier it will be when serious integration activities hit next year.

            Show
            tjenness Tim Jenness added a comment - I don't want this RFC to be seen as a starting point for dropping Python 2 everywhere. If we all agree that in operations we are going to be using Python 3 then it makes sense that we use Python 3 for integration and test activities. The earlier we do that for our internal testing the easier it will be when serious integration activities hit next year.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            While I agree we want to migrate to Python 3 eventually, I hope we don't need independent softwares to be in Python 3 in F17. For example, Pegasus, one Workflow management software of interest, is not fully Python 3 compatible yet (work in progress to make it Python 3 compatible, AFAIK). But practically I don't think Pegasus needs to be Python 3 for me to run the processing using our Python 3 stack. Technically a workflow management software does not even need to know Python to run CmdLineTasks and manage the workflow (assuming limited interactions with the stack); I don't know how the SuperTask interface will be though.

            Maybe I misunderstood the scope: do you mean using our Python 3 stack in any test reprocessing, rather than requiring every software in Python 3?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - While I agree we want to migrate to Python 3 eventually, I hope we don't need independent softwares to be in Python 3 in F17. For example, Pegasus, one Workflow management software of interest, is not fully Python 3 compatible yet (work in progress to make it Python 3 compatible, AFAIK). But practically I don't think Pegasus needs to be Python 3 for me to run the processing using our Python 3 stack. Technically a workflow management software does not even need to know Python to run CmdLineTasks and manage the workflow (assuming limited interactions with the stack); I don't know how the SuperTask interface will be though. Maybe I misunderstood the scope: do you mean using our Python 3 stack in any test reprocessing, rather than requiring every software in Python 3?
            Hide
            tjenness Tim Jenness added a comment -

            We can't use Python 2 in operations anywhere. We need to be working towards python 3 everywhere as we develop and test all our code and we need to be aware of porting timescales from external code we rely on. If pegasus doesn't yet support python3 then we can't run it with python 3 next cycle but we have to be planning to test its python3 support as soon as it comes and we have to be worrying if we are well into 2018 and Pegasus still hasn't got a version working with python 3. We should have a risk register entry listing all the code we are using that relies on python2 along with timelines for when python3 support will be available along with a plan for what to do if they are not going to be using python 3 before commissioning starts.

            All code we have control over should use python3. The python2-only codebase should be shrinking, not growing.

            Show
            tjenness Tim Jenness added a comment - We can't use Python 2 in operations anywhere. We need to be working towards python 3 everywhere as we develop and test all our code and we need to be aware of porting timescales from external code we rely on. If pegasus doesn't yet support python3 then we can't run it with python 3 next cycle but we have to be planning to test its python3 support as soon as it comes and we have to be worrying if we are well into 2018 and Pegasus still hasn't got a version working with python 3. We should have a risk register entry listing all the code we are using that relies on python2 along with timelines for when python3 support will be available along with a plan for what to do if they are not going to be using python 3 before commissioning starts. All code we have control over should use python3. The python2-only codebase should be shrinking, not growing.
            Hide
            tjenness Tim Jenness added a comment -

            I am adopting the following plan:

            • Plan for all remaining DM software (including internal administrative scripts) being compatible with Python 3 by January 2018.
            • Where Python 3 is supported, use Python 3 for integration and testing and data challenges.
            • For the Friday hack session at the summer All Hands, focus on porting packages to Python 3. In particular work with Dave Mills and Michael Reuter on porting SAL to Python 3 (and possibly Pybind11 and ndarray with help from Pim Schellart [X]).

            I will discuss with Wil O'Mullane and Frossie Economou as to the best place to document this (LDM-503 vs individual test specifications).

            Show
            tjenness Tim Jenness added a comment - I am adopting the following plan: Plan for all remaining DM software (including internal administrative scripts) being compatible with Python 3 by January 2018. Where Python 3 is supported, use Python 3 for integration and testing and data challenges. For the Friday hack session at the summer All Hands, focus on porting packages to Python 3. In particular work with Dave Mills and Michael Reuter on porting SAL to Python 3 (and possibly Pybind11 and ndarray with help from Pim Schellart [X] ). I will discuss with Wil O'Mullane and Frossie Economou as to the best place to document this (LDM-503 vs individual test specifications).

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Donald Petravick, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Mario Juric, Paul Price, Pim Schellart [X] (Inactive), Tim Jenness
              Votes:
              2 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins Builds

                  No builds found.