Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3904

meas_astrom update for pytest and SIP test can fail if tests run all in one process

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None
    • Team:
      Alert Production

      Description

      When running the meas_astrom test suite with nose the following error occurs:

      ======================================================================
      ERROR: testBigXy0 (createWcsWithSip.CreateWcsWithSipCase)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 81, in testBigXy0
          res = self.astrom.determineWcs2(cat, bbox=bbox)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 436, in determineWcs2
          wcs,qa = self.getBlindWcsSolution(sourceCat, **kwargs)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 524, in getBlindWcsSolution
          raise RuntimeError("Unable to match sources with catalog.")
      RuntimeError: Unable to match sources with catalog.
      -------------------- >> begin captured stdout << ---------------------
      Catalog size 186
       
      --------------------- >> end captured stdout << ----------------------
       
      ======================================================================
      ERROR: testLinearXDistort (createWcsWithSip.CreateWcsWithSipCase)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 98, in testLinearXDistort
          self.singleTestInstance(self.filename, distort.linearXDistort)
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 109, in singleTestInstance
          cat = self.loadCatalogue(self.filename)
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 165, in loadCatalogue
          res = self.astrom.determineWcs2(cat, bbox=bbox)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 436, in determineWcs2
          wcs,qa = self.getBlindWcsSolution(sourceCat, **kwargs)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 524, in getBlindWcsSolution
          raise RuntimeError("Unable to match sources with catalog.")
      RuntimeError: Unable to match sources with catalog.
      -------------------- >> begin captured stdout << ---------------------
      linearXDistort
       
      --------------------- >> end captured stdout << ----------------------
       
      ======================================================================
      ERROR: testLinearYDistort (createWcsWithSip.CreateWcsWithSipCase)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 102, in testLinearYDistort
          self.singleTestInstance(self.filename, distort.linearYDistort)
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 109, in singleTestInstance
          cat = self.loadCatalogue(self.filename)
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 165, in loadCatalogue
          res = self.astrom.determineWcs2(cat, bbox=bbox)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 436, in determineWcs2
          wcs,qa = self.getBlindWcsSolution(sourceCat, **kwargs)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 524, in getBlindWcsSolution
          raise RuntimeError("Unable to match sources with catalog.")
      RuntimeError: Unable to match sources with catalog.
      -------------------- >> begin captured stdout << ---------------------
      linearYDistort
       
      --------------------- >> end captured stdout << ----------------------
       
      ======================================================================
      ERROR: testQuadraticDistort (createWcsWithSip.CreateWcsWithSipCase)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 106, in testQuadraticDistort
          self.singleTestInstance(self.filename, distort.linearYDistort)
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 109, in singleTestInstance
          cat = self.loadCatalogue(self.filename)
        File "/Users/timj/work/lsstsw/src/meas_astrom/tests/createWcsWithSip.py", line 165, in loadCatalogue
          res = self.astrom.determineWcs2(cat, bbox=bbox)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 436, in determineWcs2
          wcs,qa = self.getBlindWcsSolution(sourceCat, **kwargs)
        File "/Users/timj/work/lsstsw/src/meas_astrom/python/lsst/meas/astrom/anetBasicAstrometry.py", line 524, in getBlindWcsSolution
          raise RuntimeError("Unable to match sources with catalog.")
      RuntimeError: Unable to match sources with catalog.
      -------------------- >> begin captured stdout << ---------------------
      linearQuadraticDistort
       
      --------------------- >> end captured stdout << ----------------------
       
      ----------------------------------------------------------------------
      Ran 77 tests in 96.588s
       
      FAILED (SKIP=10, errors=4)
      

      (This requires the corresponding patches from the DM-3901 ticket branch to have been applied).

      Running in subprocess with:

      nosetests --processes=4 --process-timeout=240 tests/*.py
      

      leads to all tests passing. As does running just the createWcsWithSip.py test on its own with nose.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            A bit more detail, (assuming you are testing on tickets/DM-3901 branch) I can get the tests to fail simply by first testing openFiles.py:

            py.test tests/openFiles.py tests/createWcsWithSip.py
            

            Other prior tests can also trigger the problem (but the lsf*.py tests don't trigger it). nosetests has the same issue. This really does seem to be indicating some internal memory corruption.

            Tests pass with nose if createWcsWithSip.py is run first. With a similar test order py.test fails because of a file descriptor leak by the time testMultiIndex.py executes.

            Show
            tjenness Tim Jenness added a comment - A bit more detail, (assuming you are testing on tickets/ DM-3901 branch) I can get the tests to fail simply by first testing openFiles.py : py.test tests/openFiles.py tests/createWcsWithSip.py Other prior tests can also trigger the problem (but the lsf*.py tests don't trigger it). nosetests has the same issue. This really does seem to be indicating some internal memory corruption. Tests pass with nose if createWcsWithSip.py is run first. With a similar test order py.test fails because of a file descriptor leak by the time testMultiIndex.py executes.
            Hide
            tjenness Tim Jenness added a comment -

            I see that some of these tests set an environment variable ASTROMETRY_NET_DATA_DIR. My current guess is that this environment variable contaminates the other tests and needs to be cleared in a tearDown step (or else we ensure that it is set correctly by each test).

            Show
            tjenness Tim Jenness added a comment - I see that some of these tests set an environment variable ASTROMETRY_NET_DATA_DIR. My current guess is that this environment variable contaminates the other tests and needs to be cleared in a tearDown step (or else we ensure that it is set correctly by each test).
            Hide
            rowen Russell Owen added a comment - - edited

            Thanks for offering to review this.

            Two areas of possible overreach:

            • I ran autopep8 on all code
            • I changed some imports to import specific symbols instead of using a namespace abbreviation (I also changed all imports of lsst.utils.tests to import lsst.utils.tests but I think that is what is wanted for uniformity, so I don't think it is overreach).
            Show
            rowen Russell Owen added a comment - - edited Thanks for offering to review this. Two areas of possible overreach: I ran autopep8 on all code I changed some imports to import specific symbols instead of using a namespace abbreviation (I also changed all imports of lsst.utils.tests to import lsst.utils.tests but I think that is what is wanted for uniformity, so I don't think it is overreach).
            Show
            rowen Russell Owen added a comment - Jenkins CI passed: https://ci.lsst.codes/job/stack-os-matrix/compiler=gcc,label=centos-7,python=py2/14446//console
            Hide
            tjenness Tim Jenness added a comment -

            I fully stand behind the changes to the test imports. I'm also happy with the pep8 fixes as we were going to have to do them at some point.

            Show
            tjenness Tim Jenness added a comment - I fully stand behind the changes to the test imports. I'm also happy with the pep8 fixes as we were going to have to do them at some point.
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            Looks good. No comments.

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - Looks good. No comments.
            Hide
            fred3m Fred Moolekamp added a comment -

            There were a couple of ##'s that autopep8 removed. Other than that it looks good.

            Show
            fred3m Fred Moolekamp added a comment - There were a couple of ##'s that autopep8 removed. Other than that it looks good.
            Hide
            rowen Russell Owen added a comment -

            Fred Moolekamp Thank you for catching the ## to # error. I found it in fitTanSipWcs.py and fixed it.

            Show
            rowen Russell Owen added a comment - Fred Moolekamp Thank you for catching the ## to # error. I found it in fitTanSipWcs.py and fixed it.

              People

              • Assignee:
                rowen Russell Owen
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                Fred Moolekamp
                Watchers:
                Fred Moolekamp, John Swinbank, Lauren MacArthur, Pim Schellart [X] (Inactive), Russell Owen, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel