Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13316

AST errors when using multiprocessing to return astshim objects

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: astshim
    • Labels:
      None

      Description

      Using the Python multiprocessing module to return astshim objects causes AST errors. Here is a simple example:

      import multiprocessing
      import astshim as ast
       
       
      class PickleableUnitMap(ast.UnitMap):
          def __reduce__(self):
              return (ast.UnitMap, (self.nIn,))
       
       
      if __name__ == "__main__":
          numProcesses = 1
          params = [1]*numProcesses
          pool = multiprocessing.Pool(processes=numProcesses)
          pool.map_async(PickleableUnitMap, params).get()
          pool.close()
          pool.join()
      

      Running this results in the following error:

      !! AST: Error at line 41 in file include/astshim/detail/utils.h.
      !  Invalid Object pointer given (value is 389122).
      !  This pointer is currently owned by another thread (possible programming error).
      

      When I add diagnostic print statements to astshim and enable pointer watchpoints in a newer version of starlink_ast I see the following, which seems to show that the AST object pointers are being allocated and deallocated in the correct process; thread 0 owns the pointer, but the handle property shows thread 1, which is why AST reports an error:

      Main process ID 63744
      Object(0x5f101) of type UnitMap in process 63745
      astAnnul(0x5f101) in process 63745
      Object(0x5f101) of type UnitMap in process 63744
      astAnnul(0x5f101) in process 63744
      !! AST: Error at line 46 in file include/astshim/detail/utils.h.
      !  Invalid Object pointer given (value is 389377).
      !  This pointer is currently owned by another thread (possible programming error).
      !  Handle properties: (index:0 v:389377 c:0 t:0 i:9 cl:UnitMap) [cur. thread: 1] 
      

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            I see the same thing in pyast if I modify the build system to build with threads enabled. Steps to reproduce:

            • Insert the following before line 129 of setup.py: define_macros.append(('THREAD_SAFE', '1'))
            • Build and install in the usual way: python setup.py build and python setup.py install
            • Run the following example code (which uses a simplistic pickling scheme to avoid using any AST classes other than UnitMap):

            import multiprocessing
            import starlink.Ast as ast
             
            class PickleableUnitMap(ast.UnitMap):
                def __reduce__(self):
                    return (ast.UnitMap, (self.Nin,))
             
            if __name__ == "__main__":
                numProcesses = 1
                params = [1]*numProcesses
                pool = multiprocessing.Pool(processes=numProcesses)
                pool.map_async(PickleableUnitMap, params).get()
                pool.close()
                pool.join()
            

            Show
            rowen Russell Owen added a comment - I see the same thing in pyast if I modify the build system to build with threads enabled. Steps to reproduce: Insert the following before line 129 of setup.py: define_macros.append(('THREAD_SAFE', '1')) Build and install in the usual way: python setup.py build and python setup.py install Run the following example code (which uses a simplistic pickling scheme to avoid using any AST classes other than UnitMap): import multiprocessing import starlink.Ast as ast   class PickleableUnitMap(ast.UnitMap): def __reduce__(self): return (ast.UnitMap, (self.Nin,))   if __name__ == "__main__": numProcesses = 1 params = [1]*numProcesses pool = multiprocessing.Pool(processes=numProcesses) pool.map_async(PickleableUnitMap, params).get() pool.close() pool.join()
            Hide
            rowen Russell Owen added a comment - - edited

            It turns out that Pool.map unpickles the returned objects in a thread, then hands them to the main thread. That is why AST isees objects created in one thread and disposed of in another. (The attached file showMultiprocessingError.py demonstrates this ).

            I have not found a way to change this behavior of Pool.map, though I have asked on stackoverflow

            I also don't see an easy way to work with this in AST while keeping thread support. We can unlock objects in the unpickle thread and lock them again in the main thread, but I see no way to automate that. The unlocking would have to occur in the unpickling code, and that's easy to do, but then every unpickled object would have to be locked before it could be used.

            The only reasonable solution I see is to disable threading support in AST. That in turn would require careful use of locking when using Transform and SkyWcs objects with multithreading code.

            Show
            rowen Russell Owen added a comment - - edited It turns out that Pool.map unpickles the returned objects in a thread, then hands them to the main thread. That is why AST isees objects created in one thread and disposed of in another. (The attached file showMultiprocessingError.py demonstrates this ). I have not found a way to change this behavior of Pool.map, though I have asked on stackoverflow I also don't see an easy way to work with this in AST while keeping thread support. We can unlock objects in the unpickle thread and lock them again in the main thread, but I see no way to automate that. The unlocking would have to occur in the unpickling code, and that's easy to do, but then every unpickled object would have to be locked before it could be used. The only reasonable solution I see is to disable threading support in AST. That in turn would require careful use of locking when using Transform and SkyWcs objects with multithreading code.
            Hide
            rowen Russell Owen added a comment -

            Pull in the latest AST updates

            Disable threading support to fix this threading issue

            Add a unit test to astshim for this threading issue

            Pull requests:

            AST: https://github.com/lsst/starlink_ast/pull/13

            astshim: https://github.com/lsst/astshim/pull/39

             

            Show
            rowen Russell Owen added a comment - Pull in the latest AST updates Disable threading support to fix this threading issue Add a unit test to astshim for this threading issue Pull requests: AST:  https://github.com/lsst/starlink_ast/pull/13 astshim: https://github.com/lsst/astshim/pull/39  
            Hide
            tjenness Tim Jenness added a comment -

            The new test looks ok to me.

            Show
            tjenness Tim Jenness added a comment - The new test looks ok to me.

              People

              • Assignee:
                rowen Russell Owen
                Reporter:
                rowen Russell Owen
                Reviewers:
                Tim Jenness
                Watchers:
                Jim Bosch, Paul Price, Russell Owen, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel