This is somewhat of an introduction to Pykka, a Python library implementing the actor model, which I’ve been working on now and then since November. And it’s still just 300 lines of code!

The goal of the actor model is to make it easier to develop concurrent programs by removing all shared state and use messaging for all communication. Removing shared state and doing lots of messaging doesn’t make anything easier by itself, but it avoids common problems in concurrent programs like proper lock usage and data corruption (due to the absence of proper lock usage). Also, I believe the actor model makes it easier to reason about concurrent programs, which is a nice property since software development often feels like 10% development and 91% debugging.

The goal of Pykka is to implement the actor model for Python as well as building useful concurrency abstractions on top of the actor model, most notably Pykka’s actor proxies. Actor proxies let you call the actor’s methods and read/write to the actor’s public attributes through a regular API, as opposed to passing messages using the usual send_one_way(message) and send_request_reply(message, block, timeout) methods. Accessing the actor’s public attributes like this may sound like the reintroduction of shared state and not in line with the actor model, but the actor proxy has no more access to the actor than anyone with a regular reference to the running actor has: it can send messages to the actor and patiently wait for an answer. The request for reading or writing to the attribute is processed like any other message to the actor: one at the time. The only difference between a regular object API and the API of the actor proxy is that all attribute reads and method calls on the proxy returns futures. You can either just get() the future right away, to block until the result is available and effectively serialize the execution your program, or you can pass the future around your program, delaying get() until the moment where you really need the future’s encapsulated value.

Currently Pykka has two actor implementations. One is based on threads, with one thread per actor. The other is based on gevent, with one lightweight greenlet per actor. In the gevent variant, all actors share a single thread, but are scheduled by a libevent loop. libevent is a cross-platform library wrapping whatever event mechanism that is the most efficient one available on your platform. libevent is also used by the popular Node.js framework. The big disadvantage of the gevent actors are that they don’t play well with non-gevent threads. Sometimes you can’t convert all threads in your application to actors, especially if those threads are embedded in support libraries you use, like GStreamer and libspotify, as is the case for my other current project, the Mopidy music server. For those cases, the threading-based actors are there to help you.

For some examples and more detailed descriptions and documentation, check out Pykka’s documentation.

Porting to Python 3

gevent isn’t available under Python 3 yet, but it seems like a port of gevent to Python 3 is being worked on. The upcoming 0.12 release of Pykka brings Python 3 support to the thread-based actor implementation.

Porting Pykka to Python 3 was no large feat:

Pykka is a quite new project, mainly made to scatch my own itch (aren’t all projects?). I know some people have been playing with it, but I’m not aware of anybody other than myself using it for a “real” project yet. Thus, I’m quite lenient about changing the API, at least in minor ways. The current goal is to make the library strike a nice balance between being useful and consistent. If I see ways to improve the API, I mostly just do it, but make sure to increase the version number in the next release. That’s how I got to 0.12 in four months. ;-) To keep overhead low, I’ve not yet started to maintain a changelog or migration documentation outside the git log. If you start using Pykka for anything more than an hour of play, please notify me, and I’ll straighten up. My point is, not having to maintain backwards compatability simplifies the world greatly. That said, I don’t think I’ve changed any of Pykka’s API to add Python 3 support.

As my main/original target for Pykka was to use it for Mopidy, which requires 2.6+, Pykka also requires Python 2.6+. 2.6 brings e.g. with statements, which are nice when working with locks. All the three latest Ubuntu releases defaults to 2.6, and so does OS X as of Snow Leopard. (Heck, some Linux distributions like Arch Linux are already using Python 3 as the default.) If you increase the requirement to Python 2.7 (a bit early yet, I think) you can even use lots of the new modules from Python 3 which has been backported to Python 2.x. Not having to support older Python versions makes it easier to write idiomatic Python code that will work mostly unchanged on Python 3. Keep the range of supported versions as narrow as possible, gradually getting rid of support for older Python versions, while not excluding users of any recent OS distribution.

Given this setting, three things have been really helpful for porting to and supporting a project on Python 3:

• Good test coverage. Pykka is currently at 100% line coverage, and I believe the branch coverage is quite all right too. 1.2 seconds after running the tests I’m confident that everything works. That confidence is a great feeling. The tools I’ve used is unittest (not unittest2, yet), the nosetests test runner, and the nose coverage plugin.

• Have all the Python version you target available on your system. I recently upgraded to the alpha version of Ubuntu 11.04, which works nicely so far. Ubuntu 11.04 changes the default Python version from 2.6.6 to 2.7.1, but more importantly: it brings Python 3.2 to the table. In other words Python 2.6, 2.7, 3.1, and 3.2 on a single system is just sudo apt-get install python-all python3-all away.

• A test runner which makes testing on multiple Python versions a piece of cake: tox. Thanks to tox, it is actually 6 characters less effort to run the tests on four Python versions instead of running them one Python version (tox vs nosetests). Of course, it takes more than four times as long to run, but when the time for a single Python version is 1.2 seconds, I’ll survive the wait, and I’ll even run the tests on all targeted versions before every commit.

Porting to Python 3 is not just a one time effort – an effort which probably is quite small for many idiomatic and well-tested Python projects – but also an ongoing effort. There is little gain in porting if your going to break the Python 3 support with your next commit. Having confidence in your tests and a couple of nice tools is all you need to support multiple versions in parallel.

If you’re not writing tests, I’m sorry. I can’t help you.

Update 2011-03-30: Pykka 0.12 has now been released.