Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7600

Pytest and Flake8 integration with CI and sconsUtils

    XMLWordPrintable

Details

    • Epic
    • Status: Done
    • Resolution: Done
    • None
    • None
    • f16-pytest-flake8
    • 15
    • 1.02C.10.02
    • SQuaRE
    • Fall 2016

    Description

      Integrate pytest and flake8 into sconsUtils and integrate it into CI.

      This includes turning master protection with merge gatekeeped on being up to date with master on Architecture request.

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment -

            See also DM-5637

            tjenness Tim Jenness added a comment - See also DM-5637
            tjenness Tim Jenness added a comment - - edited

            I've added the open pytest porting tickets to this as blockers. Note that they don't block the work sorting out integrating pytest into Jenkins (you can just ask to build packages that are compliant), they block the deployment of pytest support to the live CI system. Also, danielsf, ljones how close is lsst_sims to being pytest compliant? I have no insight into the sims pytest progress.

            tjenness Tim Jenness added a comment - - edited I've added the open pytest porting tickets to this as blockers. Note that they don't block the work sorting out integrating pytest into Jenkins (you can just ask to build packages that are compliant), they block the deployment of pytest support to the live CI system. Also, danielsf , ljones how close is lsst_sims to being pytest compliant? I have no insight into the sims pytest progress.
            danielsf Scott Daniel added a comment -

            tjenness Sims is 100% pytest compliant.

            danielsf Scott Daniel added a comment - tjenness Sims is 100% pytest compliant.
            tjenness Tim Jenness added a comment -

            jmatt some thoughts/options on flake8 and pytest integration with CI.

            flake8:

            • Add travis jobs to every package so that github pull requests trigger a report. I think this only works if the code is already 100% flake8-clean otherwise running it but not failing the travis run is not helpful. Can travis attach the flake8 report as a comment on a github Pull Request?
            • Should Jenkins run flake8 as part of a standard lsstsw build? What happens to the reports? Should it be a separate Jenkins flake8 build? Will anyone notice if that build generates errors?
            • Jenkins should only run flake8 on non-default branches. Running flake8 on master is irrelevant as the person submitting the job won't be able to fix anything.

            pytest

            • Should sconsUtils run pytest tests/testme.py rather than running python tests/testme.py? This would generate one JUnit file per test but is the easiest modification.
            • Should sconsUtils run pytest tests/test1.py tests/test2.py ... (so all the tests together).
            • Can Jenkins collate JUnit XML files generated when that package has built? Can it collate coverage reports?
            • If sconsUtils is running the tests we need to ensure that people running rebuild get the test output and not just a JUnit XML file.
            • Should sconsUtils have no changes but have lsst-build run a pytest afterburner? This should run even if sconsUtils noted there were problems.
            • Should Jenkins have a mode to run all tests from all built packages in one process after a build? Can we generate coverage reports from that?
            tjenness Tim Jenness added a comment - jmatt some thoughts/options on flake8 and pytest integration with CI. flake8 : Add travis jobs to every package so that github pull requests trigger a report. I think this only works if the code is already 100% flake8-clean otherwise running it but not failing the travis run is not helpful. Can travis attach the flake8 report as a comment on a github Pull Request? Should Jenkins run flake8 as part of a standard lsstsw build? What happens to the reports? Should it be a separate Jenkins flake8 build? Will anyone notice if that build generates errors? Jenkins should only run flake8 on non-default branches. Running flake8 on master is irrelevant as the person submitting the job won't be able to fix anything. pytest Should sconsUtils run pytest tests/testme.py rather than running python tests/testme.py ? This would generate one JUnit file per test but is the easiest modification. Should sconsUtils run pytest tests/test1.py tests/test2.py ... (so all the tests together). Can Jenkins collate JUnit XML files generated when that package has built? Can it collate coverage reports? If sconsUtils is running the tests we need to ensure that people running rebuild get the test output and not just a JUnit XML file. Should sconsUtils have no changes but have lsst-build run a pytest afterburner? This should run even if sconsUtils noted there were problems. Should Jenkins have a mode to run all tests from all built packages in one process after a build? Can we generate coverage reports from that?
            tjenness Tim Jenness added a comment -

            I've added a link to DM-743 as it's directly related to this work and probably should be closed when we have CI integration with flake8.

            tjenness Tim Jenness added a comment - I've added a link to DM-743 as it's directly related to this work and probably should be closed when we have CI integration with flake8.
            tjenness Tim Jenness added a comment -

            Also, RFC-229 might simplify pytest integration because if we do rename then the test target could be to run pytest without arguments rather than trying to get a specific list of test files from scons (test binaries, if present, would still have to be built of course).

            tjenness Tim Jenness added a comment - Also, RFC-229 might simplify pytest integration because if we do rename then the test target could be to run pytest without arguments rather than trying to get a specific list of test files from scons (test binaries, if present, would still have to be built of course).
            frossie Frossie Economou added a comment - - edited

            This epic initially was going to cover both pytest and flake8 intergration. We pursued flake8 integration and then after consultation with the DM system engineer we decided to do the work to gatekeep merges on successful flake8 CI which in turn required providing a tool that can work on repositories in bulk to manipulate branch protection, for example in order to require that a branch is rebased before it can be merged. The code for this is in https://github.com/lsst-sqre/sqre-gtf

            The pytest work will be scheduled at a future date.

            frossie Frossie Economou added a comment - - edited This epic initially was going to cover both pytest and flake8 intergration. We pursued flake8 integration and then after consultation with the DM system engineer we decided to do the work to gatekeep merges on successful flake8 CI which in turn required providing a tool that can work on repositories in bulk to manipulate branch protection, for example in order to require that a branch is rebased before it can be merged. The code for this is in https://github.com/lsst-sqre/sqre-gtf The pytest work will be scheduled at a future date.

            People

              jmatt J Matt Peterson [X] (Inactive)
              jmatt J Matt Peterson [X] (Inactive)
              Frossie Economou
              Fritz Mueller, Frossie Economou, J Matt Peterson [X] (Inactive), John Parejko, Scott Daniel, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.