Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7600

Pytest and Flake8 integration with CI and sconsUtils

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Epic Name:
      f16-pytest-flake8
    • Story Points:
      15
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Fall 2016

      Description

      Integrate pytest and flake8 into sconsUtils and integrate it into CI.

      This includes turning master protection with merge gatekeeped on being up to date with master on Architecture request.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            See also DM-5637

            Show
            tjenness Tim Jenness added a comment - See also DM-5637
            Hide
            tjenness Tim Jenness added a comment - - edited

            I've added the open pytest porting tickets to this as blockers. Note that they don't block the work sorting out integrating pytest into Jenkins (you can just ask to build packages that are compliant), they block the deployment of pytest support to the live CI system. Also, Scott Daniel, Lynne Jones how close is lsst_sims to being pytest compliant? I have no insight into the sims pytest progress.

            Show
            tjenness Tim Jenness added a comment - - edited I've added the open pytest porting tickets to this as blockers. Note that they don't block the work sorting out integrating pytest into Jenkins (you can just ask to build packages that are compliant), they block the deployment of pytest support to the live CI system. Also, Scott Daniel , Lynne Jones how close is lsst_sims to being pytest compliant? I have no insight into the sims pytest progress.
            Hide
            danielsf Scott Daniel added a comment -

            Tim Jenness Sims is 100% pytest compliant.

            Show
            danielsf Scott Daniel added a comment - Tim Jenness Sims is 100% pytest compliant.
            Hide
            tjenness Tim Jenness added a comment -

            J Matt Peterson [X] some thoughts/options on flake8 and pytest integration with CI.

            flake8:

            • Add travis jobs to every package so that github pull requests trigger a report. I think this only works if the code is already 100% flake8-clean otherwise running it but not failing the travis run is not helpful. Can travis attach the flake8 report as a comment on a github Pull Request?
            • Should Jenkins run flake8 as part of a standard lsstsw build? What happens to the reports? Should it be a separate Jenkins flake8 build? Will anyone notice if that build generates errors?
            • Jenkins should only run flake8 on non-default branches. Running flake8 on master is irrelevant as the person submitting the job won't be able to fix anything.

            pytest

            • Should sconsUtils run pytest tests/testme.py rather than running python tests/testme.py? This would generate one JUnit file per test but is the easiest modification.
            • Should sconsUtils run pytest tests/test1.py tests/test2.py ... (so all the tests together).
            • Can Jenkins collate JUnit XML files generated when that package has built? Can it collate coverage reports?
            • If sconsUtils is running the tests we need to ensure that people running rebuild get the test output and not just a JUnit XML file.
            • Should sconsUtils have no changes but have lsst-build run a pytest afterburner? This should run even if sconsUtils noted there were problems.
            • Should Jenkins have a mode to run all tests from all built packages in one process after a build? Can we generate coverage reports from that?
            Show
            tjenness Tim Jenness added a comment - J Matt Peterson [X] some thoughts/options on flake8 and pytest integration with CI. flake8 : Add travis jobs to every package so that github pull requests trigger a report. I think this only works if the code is already 100% flake8-clean otherwise running it but not failing the travis run is not helpful. Can travis attach the flake8 report as a comment on a github Pull Request? Should Jenkins run flake8 as part of a standard lsstsw build? What happens to the reports? Should it be a separate Jenkins flake8 build? Will anyone notice if that build generates errors? Jenkins should only run flake8 on non-default branches. Running flake8 on master is irrelevant as the person submitting the job won't be able to fix anything. pytest Should sconsUtils run pytest tests/testme.py rather than running python tests/testme.py ? This would generate one JUnit file per test but is the easiest modification. Should sconsUtils run pytest tests/test1.py tests/test2.py ... (so all the tests together). Can Jenkins collate JUnit XML files generated when that package has built? Can it collate coverage reports? If sconsUtils is running the tests we need to ensure that people running rebuild get the test output and not just a JUnit XML file. Should sconsUtils have no changes but have lsst-build run a pytest afterburner? This should run even if sconsUtils noted there were problems. Should Jenkins have a mode to run all tests from all built packages in one process after a build? Can we generate coverage reports from that?
            Hide
            tjenness Tim Jenness added a comment -

            I've added a link to DM-743 as it's directly related to this work and probably should be closed when we have CI integration with flake8.

            Show
            tjenness Tim Jenness added a comment - I've added a link to DM-743 as it's directly related to this work and probably should be closed when we have CI integration with flake8.
            Hide
            tjenness Tim Jenness added a comment -

            Also, RFC-229 might simplify pytest integration because if we do rename then the test target could be to run pytest without arguments rather than trying to get a specific list of test files from scons (test binaries, if present, would still have to be built of course).

            Show
            tjenness Tim Jenness added a comment - Also, RFC-229 might simplify pytest integration because if we do rename then the test target could be to run pytest without arguments rather than trying to get a specific list of test files from scons (test binaries, if present, would still have to be built of course).
            Hide
            frossie Frossie Economou added a comment - - edited

            This epic initially was going to cover both pytest and flake8 intergration. We pursued flake8 integration and then after consultation with the DM system engineer we decided to do the work to gatekeep merges on successful flake8 CI which in turn required providing a tool that can work on repositories in bulk to manipulate branch protection, for example in order to require that a branch is rebased before it can be merged. The code for this is in https://github.com/lsst-sqre/sqre-gtf

            The pytest work will be scheduled at a future date.

            Show
            frossie Frossie Economou added a comment - - edited This epic initially was going to cover both pytest and flake8 intergration. We pursued flake8 integration and then after consultation with the DM system engineer we decided to do the work to gatekeep merges on successful flake8 CI which in turn required providing a tool that can work on repositories in bulk to manipulate branch protection, for example in order to require that a branch is rebased before it can be merged. The code for this is in https://github.com/lsst-sqre/sqre-gtf The pytest work will be scheduled at a future date.

              People

              • Assignee:
                jmatt J Matt Peterson [X] (Inactive)
                Reporter:
                jmatt J Matt Peterson [X] (Inactive)
                Reviewers:
                Frossie Economou
                Watchers:
                Fritz Mueller, Frossie Economou, J Matt Peterson [X] (Inactive), John Parejko, Scott Daniel, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel