Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28109

Test failure in obs_base test_cameraMapper.Mapper2TestCase

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_base
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      AP S21-1 (December)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      This seems like it has to be caused by DM-27178. But I see Jenkins passed on that branch at 2020-12-16T04:11Z (a force-push thereafter did not significantly change the code), and the nightly lsst_distrib Jenkins build also passed.

      The weird thing is that (on lsst-devl02) pytest seems to succeed, as does scons -j 1, but scons -j 8 fails. 20 runs of pytest -n 8 tests/test_cameraMapper.py separated by rm -rf .pytest_cache gives 2 passes and 18 failures. 20 similar runs with -n 4 gives 14 passes and only 6 failures. Interestingly, -n 5 gave 20 failures, and -n 3 gave 19.

      Is this a race condition having to do with the Filter singleton?

      But the pytest workers run in separate processes, so I don't see how that happens.

        Attachments

          Activity

          Hide
          krzys Krzysztof Findeisen added a comment - - edited

          It does seem plausible that this could be a conflict between MinCam and some real instrument. Constructing any Instrument resets the filters, and I hadn't realized at the time that I wrote it that testStandardizeFiltersFilterNoDefs really does need the filters to be registered (testStandardizeFiltersFilterDefs should still be safe, as it should not exercise any code paths that use Filter).

          Given that we presumably need to merge this today, should I disable MinCam (and both tests that depend on it) entirely, or just testStandardizeFiltersFilterNoDefs? Creating a test Instrument was a bit of a risk in the first place, I just thought (based on scons -j 6) that it had worked out.

          Show
          krzys Krzysztof Findeisen added a comment - - edited It does seem plausible that this could be a conflict between MinCam and some real instrument. Constructing any Instrument resets the filters, and I hadn't realized at the time that I wrote it that testStandardizeFiltersFilterNoDefs really does need the filters to be registered ( testStandardizeFiltersFilterDefs should still be safe, as it should not exercise any code paths that use Filter ). Given that we presumably need to merge this today, should I disable MinCam (and both tests that depend on it) entirely, or just testStandardizeFiltersFilterNoDefs ? Creating a test Instrument was a bit of a risk in the first place, I just thought (based on scons -j 6 ) that it had worked out.
          Hide
          krzys Krzysztof Findeisen added a comment - - edited

          Wait, I know what the problem is: if testStandardizeFiltersNoDefs is run before all tests that use MinMapper2, then there is no active set of filter definitions. That would explain why even Filter('r') (which is supported by e.g. obs_test) returns a "no such filter".

          So I could patch the test so that it guarantees a filter definition. In tests this makes me unable to reproduce the bug as described above, but there might still be other races between Instruments, as speculated above.

          Show
          krzys Krzysztof Findeisen added a comment - - edited Wait, I know what the problem is: if testStandardizeFiltersNoDefs is run before all tests that use MinMapper2 , then there is no active set of filter definitions. That would explain why even Filter('r') (which is supported by e.g. obs_test ) returns a "no such filter". So I could patch the test so that it guarantees a filter definition. In tests this makes me unable to reproduce the bug as described above, but there might still be other races between Instruments , as speculated above.
          Hide
          ktl Kian-Tat Lim added a comment -

          A test-to-test dependency within the same file makes sense and is problematic. But as long as there is no other real Instrument in the same test file, I think you're OK even running in parallel; I don't think that afw singletons are shared between those executions.

          Show
          ktl Kian-Tat Lim added a comment - A test-to-test dependency within the same file makes sense and is problematic. But as long as there is no other real Instrument in the same test file, I think you're OK even running in parallel; I don't think that afw singletons are shared between those executions.
          Hide
          krzys Krzysztof Findeisen added a comment -

          Kian-Tat Lim, can you check if my solution meets with your approval?

          Show
          krzys Krzysztof Findeisen added a comment - Kian-Tat Lim , can you check if my solution meets with your approval?
          Hide
          ktl Kian-Tat Lim added a comment -

          Looks reasonable. Thanks!

          Show
          ktl Kian-Tat Lim added a comment - Looks reasonable. Thanks!

            People

            Assignee:
            krzys Krzysztof Findeisen
            Reporter:
            ktl Kian-Tat Lim
            Reviewers:
            Kian-Tat Lim
            Watchers:
            Kian-Tat Lim, Krzysztof Findeisen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.