multiprocessing.Connection is not free

I’ve previously written about how to wrap multiprocessing.Connection objects in a class that makes Connection objects picklable, and thus transferable over another Connection. That approach is still valid, but don’t use it–or Connection objects at all–for lots of use-once-and-throw-away connections.

For Pykka 0.9 and 0.10 I used wrapped Connection objects for implementing futures. It worked great until I tried using it for my ongoing rewrite of Mopidy to use Pykka actors. As soon as I had updated the majority of Mopidy’s tests to use actors, the test suite started failing after a given number of tests with the error “too many open files”. This error is due to multiprocessing using UNIX sockets in /tmp to implement it’s connections. The reason for backing connections with sockets is to make them work between multiple processes, a core feature of multiprocessing, but not a feature that I need.

In Pykka 0.11 I’ve rewritten the ThreadingFuture implementation from using multiprocessing.Connection to using Queue.Queue (or queue.Queue if your are using Python 3.x). As Queue.Queue use thread’s shared memory and locks instead of sockets on disk, it will not hit the “too many open files” problem, and it should also be a bit faster: A given test class in Mopidy took 2.7-2.9 seconds to run using Pykka 0.10 and multiprocessing.Connection. The same test class took 1.8-1.9 seconds using Pykka 0.11 and Queue.Queue. That’s an improvement of about one third, and a nice side effect of fixing the “too many open files” issue.