I’ve previously written about how to wrap
multiprocessing.Connection objects in a class that makes
Connection objects picklable, and thus transferable over another
Connection. That approach is still valid, but don’t use it–or
Connection objects at all–for lots of use-once-and-throw-away connections.
For Pykka 0.9 and 0.10 I used wrapped
Connection objects for implementing futures. It worked great until I tried using it for my ongoing rewrite of Mopidy to use Pykka actors. As soon as I had updated the majority of Mopidy’s tests to use actors, the test suite started failing after a given number of tests with the error “too many open files”. This error is due to
multiprocessing using UNIX sockets in
/tmp to implement it’s connections. The reason for backing connections with sockets is to make them work between multiple processes, a core feature of
multiprocessing, but not a feature that I need.
In Pykka 0.11 I’ve rewritten the
ThreadingFuture implementation from using
multiprocessing.Connection to using
queue.Queue if your are using Python 3.x). As
Queue.Queue use thread’s shared memory and locks instead of sockets on disk, it will not hit the “too many open files” problem, and it should also be a bit faster: A given test class in Mopidy took 2.7-2.9 seconds to run using Pykka 0.10 and
multiprocessing.Connection. The same test class took 1.8-1.9 seconds using Pykka 0.11 and
Queue.Queue. That’s an improvement of about one third, and a nice side effect of fixing the “too many open files” issue.