Details
-
Type:
RFC
-
Status: Adopted
-
Resolution: Done
-
Component/s: DM
-
Labels:None
-
Location:this ticket
Description
sconsUtils provides fundamental functionality for building LSST packages but has a couple of issues that, if fixed, would significantly help with advanced continuous integration builds and metrics gathering. In this RFC I propose we make the following changes to sconsUtils:
- Enable EUPS-less builds. Allowing the build to run without requiring EUPS be available simplifies decomposition of the build into individual entities. This is an extension of
RFC-44. There is a bug report inDM-2769. - Run tests with py.test or nose. The tests currently write a .failed file with the output of the test. This is not parseable by standard test reporting environments. We propose to modify the test execution to use py.test or equivalent capable of writing test results in a standard form that can be understand by many tools. This will improve the readability and metrics analysis of tests (at the expense of one more python dependency). This may require some tweaks to how tests are laid out currently and account will be made of the memory test case (including running tests with valgrind enabled).
- Replace the ups/*.cfg file with a file in standard pkg_config format. This will allow for the installed packages to be queried for dependencies using standard tools and support non-EUPS thirdparty applications linking against the stack C++.
Attachments
Issue Links
- is blocked by
-
- Implemented
- is triggering
-
DM-2769 sconsUtils has a hard dependency on EUPS for both tests and installation
- Done
-
DM-3862 Stack Build, Packaging and Testing Improvements Part II
- Done
-
- Done
-
- To Do
-
- Done
-
- Done
-
- Won't Fix
- relates to
-
- Done
-
DM-3197 Upgrade scons to v2.3.5
- Done
-
DLP-336 Engineer Decomposed Stack Builds
-
- Unplanned
-
-
- Withdrawn
Activity
+1 on the RFC (esp. the first and the last points). Another benefit to using pkg-config is that it then becomes easier to integrate with system packages (i.e., if I have fftw installed on the system, I don't need to supply a .cfg for it).
Also +1 on increasing the granularity of the build. Some history: That's what lsst_build does internally, if you think of lsst_build itself as the CI system, and the topological sort was just a convenient+quick solution at the time. The design rationale for that (otherwise crazy) decision to push everything into lsst_build was to let us decouple the builds from Buildbot (keeping just the UI), so we can then smoothly transition the UI, and then ditch buildbot completely).
Bottm line: +1 on all of this.
If this is a prerequisite for the way we want to do multi-cadence integration tests, consider my objections (which were already minor) withdrawn; I was seeing this as something that might take effort away from getting integration tests running.
Well, I may have overstated the case a bit – it's not a prerequisite for multi-cadence integration tests, but overall moving to a more tool-driven build instead of a script-driven build should help the situation. We could still do multi-cadence integration without that as long as we can get persistent state (what has been called "binary build artifacts").
Consensus reached.
In addition, decomposing the build by getting rid of lsstswBuild.sh should allow us to much more easily add integration tests for the stack, including tests that are run at different cadences. I think this is one of the highest priority results. While it could be achieved in other ways, planning for the future and the synergy with the other aspects of this RFC argues for doing it this way.