Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16343

pytest-flake8 creates many forked processes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Third Party Software
    • Labels:
      None
    • Story Points:
      0.5
    • Sprint:
      Arch 2018-11-05
    • Team:
      Architecture

      Description

      Jim Bosch noticed that when he runs scons he gets many unexpected python processes created by pytest. It looks like for every pytest core an additional N processes are created where N seems to be the number of effective cores.

        Attachments

          Issue Links

          links to

          Web Link GitHub pull request

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            The problem seems to be that pytest-flake8 runs flake8 with the default value for -jobs which causes flake8 to run with whatever the multiprocessing module thinks is a good number of default cores. Since pytest-flake8 runs flake8 separately for each file this is not the right option to use and explicitly setting the value to -j 1 fixes the problem and makes testing go significantly faster.

            Show
            tjenness Tim Jenness added a comment - The problem seems to be that pytest-flake8 runs flake8 with the default value for - jobs which causes flake8 to run with whatever the multiprocessing module thinks is a good number of default cores. Since pytest-flake8 runs flake8 separately for each file this is not the right option to use and explicitly setting the value to -j 1 fixes the problem and makes testing go significantly faster.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            This patch seems to work for me and is a significant speed up improvement. I've submitted a pull request upstream.

            Show
            tjenness Tim Jenness added a comment - This patch seems to work for me and is a significant speed up improvement. I've submitted a pull request upstream.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Looks good, thanks for taking care of this!

            I'm curious why the fix is{{ -jobs 0}} rather than -jobs 1.

            Show
            jbosch Jim Bosch added a comment - Looks good, thanks for taking care of this! I'm curious why the fix is{{ - jobs 0}} rather than -jobs 1.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Merged with jobs=1 (to at least be consistent with the surrounding comment).

            Show
            tjenness Tim Jenness added a comment - Merged with jobs=1 (to at least be consistent with the surrounding comment).

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.