Paul Price added a comment - 18/Oct/16 8:51 AM This appears to be a fundamental limitation in mpi4py and python: $ python -c 'import cPickle as pickle; print len(pickle.dumps("X"*2**31, -1))' Traceback (most recent call last): File "<string>", line 1, in <module> SystemError: error return without exception set I can reproduce the error with some simple code: import mpi4py.MPI as mpi import cPickle as pickle   def main(): comm = mpi.COMM_WORLD rank = comm.Get_rank() print rank, "OK" if rank == 0: size = int(2**31 - 1) string = "X"*size print "MASTER:", len(pickle.dumps(string, -1)) else: string = None string = comm.bcast(string, root=0) print rank, len(string)   if __name__ == "__main__": main() Running this under MPI: $ mpiexec -n 2 python test.py 0 OK 1 OK MASTER: 2147483657 Traceback (most recent call last): File "test.py", line 18, in <module> main() File "test.py", line 14, in main string = comm.bcast(string, root=0) File "MPI/Comm.pyx", line 1276, in mpi4py.MPI.Comm.bcast (src/mpi4py.MPI.c:108819) File "MPI/msgpickle.pxi", line 612, in mpi4py.MPI.PyMPI_bcast (src/mpi4py.MPI.c:47005) File "MPI/msgpickle.pxi", line 119, in mpi4py.MPI.Pickle.dump (src/mpi4py.MPI.c:40840) File "MPI/msgbuffer.pxi", line 35, in mpi4py.MPI.downcast (src/mpi4py.MPI.c:29070) OverflowError: integer 2147483657 does not fit in 'int' ^C[mpiexec@tiger-sumire] Sending Ctrl-C to processes as requested [mpiexec@tiger-sumire] Press Ctrl-C again to force abort I'm going to see if I can work around this and/or reduce the amount of data that's being transferred.

Paul Price added a comment - 18/Oct/16 11:13 AM - edited I can't work around this at the MPI level because pickle apparently is limited to 32-bit integers in python 2 (this is fixed in python 3, with protocol 4; PEP 3154 ). I tried sending the pickle by chunks smaller than 2**31 bytes, but python simply can't unpickle something whose size doesn't fit into an int , so unless we come up with our own 64-bit serialisation format we can't fix things that way. I'm going to have to go the route of reducing the amount of data that's being transferred.

John Swinbank added a comment - 18/Oct/16 12:11 PM Fritz Mueller — following up on our recent discussion, note that this is the most urgent issue we're currently looking at in the (broadly defined) Task Framework. Paul is the expert on this code.

Paul Price added a comment - 18/Oct/16 2:17 PM I have a workaround that seems to work on Lauren's test case. I split the: self.comm.broadcast((tags, func, args, kwargs, context), root=self.root) to broadcast the tuple elements individually, and I changed the distribution of initial jobs: self.comm.scatter(initial, root=self.rank) to distribute the jobs directly with send rather than scatter . I'll clean this up and then submit it for review tomorrow. In the mean time, I've put the dirty version on the ticket branch in case Lauren MacArthur needs to make progress tonight.

Lauren MacArthur added a comment - 18/Oct/16 2:21 PM Thanks, Paul. I won't be rerunning until tomorrow, so I'll try with whatever state the branch is in then.

Paul Price added a comment - 18/Oct/16 5:26 PM Fritz Mueller , since you're interested, would you be willing to review this fix? price@price-laptop:~/LSST/ctrl/pool (tickets/DM-8021=) $ git sub-patch commit 9df40a36febd5bbcfd3842356097aaa3ba79860f Author: Paul Price <price@astro.princeton.edu> Date: Tue Oct 18 20:19:00 2016 -0400   Comm: add alternate version of scatter The default version apparently pickles the entire 'dataList', which can cause errors if the pickle size grows over 2^31 bytes due to fundamental problems with pickle in python 2 [1][2] (causing, e.g., "OverflowError: integer 2155421250 does not fit in 'int'"). Instead, we send the data to each slave node in turn; this reduces the pickle size. [1] http://bugs.python.org/issue11564 [2] https://www.python.org/dev/peps/pep-3154/   diff --git a/python/lsst/ctrl/pool/pool.py b/python/lsst/ctrl/pool/pool.py index 59e1277..ce3d663 100644 --- a/python/lsst/ctrl/pool/pool.py +++ b/python/lsst/ctrl/pool/pool.py @@ -300,6 +300,30 @@ class Comm(mpi.Intracomm): with PickleHolder(value): return super(Comm, self).bcast(value, root=root) + def scatter(self, dataList, root=0, tag=0): + """Scatter data across the nodes + + The default version apparently pickles the entire 'dataList', + which can cause errors if the pickle size grows over 2^31 bytes + due to fundamental problems with pickle in python 2. Instead, + we send the data to each slave node in turn; this reduces the + pickle size. + + @param dataList List of data to distribute; one per node + (including root) + @param root Index of root node + @param tag Message tag (integer) + @return Data for this node + """ + if self.Get_rank() == root: + for rank, data in enumerate(dataList): + if rank == root: + continue + self.send(data, rank, tag=tag) + return dataList[root] + else: + return self.recv(source=root, tag=tag) + def Free(self): if self._barrierComm is not None: self._barrierComm.Free()

Lauren MacArthur added a comment - 19/Oct/16 11:19 AM I can confirm that running with the current ticket branch solve the problem. Thanks Paul!

Paul Price added a comment - 20/Oct/16 11:30 AM Fritz Mueller indicated that Nate Pease [X] might be a better reviewer. Nate, would you mind taking a look at this, please?

Nate Pease [X] (Inactive) added a comment - 20/Oct/16 12:33 PM it looks good to me.

Paul Price added a comment - 20/Oct/16 12:39 PM Thanks Nate! Merged to master.