ImperialViolet

(background reading: PEP 334)

Can you believe that I'm still going on about async IO programming? Well if someone would get it right I could shutup

My current framework du jour is one I did myself based on Stackless. Yes, I've played with Twisted a lot, and I'm not a huge fan. For one, the core itself isn't 1.0 standard (the reactors still have stupid bugs where they listen on closed sockets and short circuit) and the http code is unusable in a hostile environment.

Stackless provides user-land threads and my framework is pretty standard. The main problems are that having to patch the python interpreter is a pain and there are a few parts of Stackless that I don't quite understand - mostly because the documentation isn't there.

PEP 332 promises some of same things as stackless - but in the standard CPython. Let's look at a Python generator:

def gen():
    a = 1
    yield a
    yield 2

This function returns multiple values and keeps state between invocations. The ability to keep state is very similar to user-land threads and one can easily imagine a generator which yields values from a socket. However, when the operation blocks the generator would have to yield a out-of-band value to denote this. Every use of the socket generator would then have to handle this - dragging the code quickly into the realm of the unreadable and unwritable.

This issue is very similar to error handling and we have a way to cope with this - exceptions. So the idea of PEP 334 is to allow generators to raise a SuspendIteration exception without destroying themselves. (At the moment, once a generator has raised an exception it is finished.) The SuspendIteration would carry a payload of the objects which it is blocking on.

The top of the call chain would be the IO core which would call each top-level suspended iterator (which would call others etc) until it hit a blocking IO operation and raised SuspendIteration. This would run back up the call chain and the IO core would make note of which objects (sockets etc) which that generator is blocking on. Later if those objects become ready the generator can be called again and allowed to progress.

So the first issue is that there's no way to poke an exception into the bottom of a generator. (The ability to poke a TimedOut exception is very useful.) But, so long as all the blocking objects (wrappers around sockets, Channels, Mutexes etc) pay attention to a global variable they could be made to raise a given exception.

Thus I cheer PEP 334 onwards because it could lead to a nice IO framework that works in all the pythons without patching.