Update some tests to support nose and/or py.test

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
4
• Team:
SQuaRE

Description

When sconsUtils is migrated to use nose or py.test some test scripts will need to be modified because test discovery will be slightly different and the namespace of test execution will change.

Two things to consider:

• People would still like the option of running a test as python tests/testMe.py.
• We have to work out how to run the memory test case.

Activity

Hide
Russell Owen added a comment - - edited

You raise good points about the parent class being run and about skipping tests that may not need to be skipped. It would be worth looking at the latter – if not on this ticket, then filing a new ticket.

However, I was merely suggesting that the string "afwdata isn't set up" would be better than "Skipping <foo> test case as afwdata isn't set up" because the longer version is subject to copy/paste errors which could result in lying about what is being skipped.

Show
Russell Owen added a comment - - edited You raise good points about the parent class being run and about skipping tests that may not need to be skipped. It would be worth looking at the latter – if not on this ticket, then filing a new ticket. However, I was merely suggesting that the string "afwdata isn't set up" would be better than "Skipping <foo> test case as afwdata isn't set up" because the longer version is subject to copy/paste errors which could result in lying about what is being skipped.
Hide
Tim Jenness added a comment -

Thank you. I have made the suggested changes. I have also worked out how to rework the meas_astrom tests so that they do not trigger the skips in the base class. This involved splitting the base class into two and using multiple inheritance but it seems to work well. These have now been merged. Changes to other packages will now go on separate tickets.

Show
Tim Jenness added a comment - Thank you. I have made the suggested changes. I have also worked out how to rework the meas_astrom tests so that they do not trigger the skips in the base class. This involved splitting the base class into two and using multiple inheritance but it seems to work well. These have now been merged. Changes to other packages will now go on separate tickets.
Hide
Tim Jenness added a comment -

Following up on the question from Russell Owen about memory test, the simplest option is to add this to each test file:

 class MemoryTest(utilsTests.MemoryTestCase):  pass 

There will inevitably be code duplication whilst we try to retain the ability to use the old style suite() approach as well as supporting standardized unittest test discovery.

Show
Tim Jenness added a comment - Following up on the question from Russell Owen about memory test, the simplest option is to add this to each test file: class MemoryTest(utilsTests.MemoryTestCase): pass There will inevitably be code duplication whilst we try to retain the ability to use the old style suite() approach as well as supporting standardized unittest test discovery.
Hide
Kian-Tat Lim added a comment -

Adding a MemoryTest that can be run at any time during the execution of the tests does not accomplish the goals of the MemoryTestCase, which is to check for memory leakage during execution of (all) other tests. I hadn't fully realized that one result of this ticket is to have every test execute in an independent context. If that's true, then MemoryTestCase needs to be a wrapper around every single test. I'm not sure if unittest, nose, and/or py.test provide a means for specifying such a wrapper.

Show
Kian-Tat Lim added a comment - Adding a MemoryTest that can be run at any time during the execution of the tests does not accomplish the goals of the MemoryTestCase , which is to check for memory leakage during execution of (all) other tests. I hadn't fully realized that one result of this ticket is to have every test execute in an independent context. If that's true, then MemoryTestCase needs to be a wrapper around every single test. I'm not sure if unittest , nose , and/or py.test provide a means for specifying such a wrapper.
Hide
Tim Jenness added a comment -

I think py.test is the opposite. Every test runs within the same process (unlike now where a new python process is spawned per test file). This is why DM-3904 is a problem and why DM-5164 needed a fix to reset the policy. py.test runs all the tests in sequence so the memory test can end up being run multiple times. It does seem to be true that there is no way to guarantee that the memory test will be run as the final test from each test file (when py.test uses unittest for test discovery) but in aggregate the memory test will be run multiple times if each test file includes it.

Show
Tim Jenness added a comment - I think py.test is the opposite. Every test runs within the same process (unlike now where a new python process is spawned per test file). This is why DM-3904 is a problem and why DM-5164 needed a fix to reset the policy. py.test runs all the tests in sequence so the memory test can end up being run multiple times. It does seem to be true that there is no way to guarantee that the memory test will be run as the final test from each test file (when py.test uses unittest for test discovery) but in aggregate the memory test will be run multiple times if each test file includes it.

People

• Assignee:
Tim Jenness
Reporter:
Tim Jenness
Reviewers:
Russell Owen
Watchers:
John Parejko, John Swinbank, Kian-Tat Lim, Michael Wood-Vasey, Russell Owen, Tim Jenness