Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6998

Problems with MemoryTest ordering

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: utils
    • Labels:
      None
    • Templates:
    • Story Points:
      3
    • Team:
      Architecture

      Description

      MemoryTestCase (or a derivative thereof) must be run as the last of all tests in a module in order to properly catch leaks.

      Our documentation implies, and SQR-012 states, that this can be achieved by listing it as the last test case in the file.

      This works for py.test, but not when using plain old unittest: the latter does not, so far as I can see, guarantee any sort of ordering as a matter of principle, and, in practice, it sorts things lexicographically (it uses whatever order it gets from running dir() on the test module, and I don't think that's guaranteed to be anything in particular).

      For example, consider testAstrometrySourceSelector.py. I made the following change to introduce a memory leak:

      --- a/tests/testAstrometrySourceSelector.py
      +++ b/tests/testAstrometrySourceSelector.py
      @@ -70,8 +70,9 @@ class TestAstrometrySourceSelector(lsst.utils.tests.TestCase):
               self.sourceSelector = sourceSelector.sourceSelectorRegistry['astrometry']()
       
           def tearDown(self):
      -        del self.src
      -        del self.sourceSelector
      +        pass
      +        #del self.src
      +        #del self.sourceSelector
       
           def testSelectSources_good(self):
               for i in range(5):
      

      Py.test catches it:

      $ py.test-2.7 testAstrometrySourceSelector.py
      [...]
      testAstrometrySourceSelector.py .........F
      [...]
      

      But simply running the test suite does not:

      $ python testAstrometrySourceSelector.py
      ..........
      ----------------------------------------------------------------------
      Ran 10 tests in 0.105s
      

      Rename the test case:

      @@ -144,7 +145,7 @@ def setup_module(module):
           lsst.utils.tests.init()
       
       
      -class MyMemoryTestCase(lsst.utils.tests.MemoryTestCase):
      +class xMyMemoryTestCase(lsst.utils.tests.MemoryTestCase):
           pass
       
       if __name__ == "__main__":
      

      And boom:

      $ python testAstrometrySourceSelector.py
      .........
      54 Objects leaked:
      

      Based on a very quick check, I think sconsUtils runs tests by simply invoking python, and I'm pretty sure that this is hard-wired into the muscle memory of many developers. In these cases, memory tests written following current guidelines won't be being properly executed.

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                swinbank John Swinbank
                Reviewers:
                John Swinbank
                Watchers:
                John Swinbank, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel