Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_base
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Sprint:AP S21-1 (December)
-
Team:Alert Production
-
Urgent?:No
Description
This seems like it has to be caused by DM-27178. But I see Jenkins passed on that branch at 2020-12-16T04:11Z (a force-push thereafter did not significantly change the code), and the nightly lsst_distrib Jenkins build also passed.
The weird thing is that (on lsst-devl02) pytest seems to succeed, as does scons -j 1, but scons -j 8 fails. 20 runs of pytest -n 8 tests/test_cameraMapper.py separated by rm -rf .pytest_cache gives 2 passes and 18 failures. 20 similar runs with -n 4 gives 14 passes and only 6 failures. Interestingly, -n 5 gave 20 failures, and -n 3 gave 19.
Is this a race condition having to do with the Filter singleton?
But the pytest workers run in separate processes, so I don't see how that happens.
Attachments
Activity
Wait, I know what the problem is: if testStandardizeFiltersNoDefs is run before all tests that use MinMapper2, then there is no active set of filter definitions. That would explain why even Filter('r') (which is supported by e.g. obs_test) returns a "no such filter".
So I could patch the test so that it guarantees a filter definition. In tests this makes me unable to reproduce the bug as described above, but there might still be other races between Instruments, as speculated above.
Krzysztof Findeisen
added a comment - - edited Wait, I know what the problem is: if testStandardizeFiltersNoDefs is run before all tests that use MinMapper2 , then there is no active set of filter definitions. That would explain why even Filter('r') (which is supported by e.g. obs_test ) returns a "no such filter".
So I could patch the test so that it guarantees a filter definition. In tests this makes me unable to reproduce the bug as described above, but there might still be other races between Instruments , as speculated above. A test-to-test dependency within the same file makes sense and is problematic. But as long as there is no other real Instrument in the same test file, I think you're OK even running in parallel; I don't think that afw singletons are shared between those executions.
Kian-Tat Lim, can you check if my solution meets with your approval?
Krzysztof Findeisen
added a comment - Kian-Tat Lim , can you check if my solution meets with your approval? Looks reasonable. Thanks!
It does seem plausible that this could be a conflict between MinCam and some real instrument. Constructing any Instrument resets the filters, and I hadn't realized at the time that I wrote it that testStandardizeFiltersFilterNoDefs really does need the filters to be registered (testStandardizeFiltersFilterDefs should still be safe, as it should not exercise any code paths that use Filter).
Given that we presumably need to merge this today, should I disable MinCam (and both tests that depend on it) entirely, or just testStandardizeFiltersFilterNoDefs? Creating a test Instrument was a bit of a risk in the first place, I just thought (based on scons -j 6) that it had worked out.