See also DM-5637

I've added the open pytest porting tickets to this as blockers. Note that they don't block the work sorting out integrating pytest into Jenkins (you can just ask to build packages that are compliant), they block the deployment of pytest support to the live CI system. Also, Scott Daniel, Lynne Jones how close is lsst_sims to being pytest compliant? I have no insight into the sims pytest progress.

Tim Jenness Sims is 100% pytest compliant.

J Matt Peterson [X] some thoughts/options on flake8 and pytest integration with CI.

flake8:

• Add travis jobs to every package so that github pull requests trigger a report. I think this only works if the code is already 100% flake8-clean otherwise running it but not failing the travis run is not helpful. Can travis attach the flake8 report as a comment on a github Pull Request?
• Should Jenkins run flake8 as part of a standard lsstsw build? What happens to the reports? Should it be a separate Jenkins flake8 build? Will anyone notice if that build generates errors?
• Jenkins should only run flake8 on non-default branches. Running flake8 on master is irrelevant as the person submitting the job won't be able to fix anything.

pytest

• Should sconsUtils run pytest tests/testme.py rather than running python tests/testme.py? This would generate one JUnit file per test but is the easiest modification.
• Should sconsUtils run pytest tests/test1.py tests/test2.py ... (so all the tests together).
• Can Jenkins collate JUnit XML files generated when that package has built? Can it collate coverage reports?
• If sconsUtils is running the tests we need to ensure that people running rebuild get the test output and not just a JUnit XML file.
• Should sconsUtils have no changes but have lsst-build run a pytest afterburner? This should run even if sconsUtils noted there were problems.
• Should Jenkins have a mode to run all tests from all built packages in one process after a build? Can we generate coverage reports from that?

I've added a link to DM-743 as it's directly related to this work and probably should be closed when we have CI integration with flake8.

Also, RFC-229 might simplify pytest integration because if we do rename then the test target could be to run pytest without arguments rather than trying to get a specific list of test files from scons (test binaries, if present, would still have to be built of course).

This epic initially was going to cover both pytest and flake8 intergration. We pursued flake8 integration and then after consultation with the DM system engineer we decided to do the work to gatekeep merges on successful flake8 CI which in turn required providing a tool that can work on repositories in bulk to manipulate branch protection, for example in order to require that a branch is rebased before it can be merged. The code for this is in https://github.com/lsst-sqre/sqre-gtf

The pytest work will be scheduled at a future date.