ImperialViolet

This is a follow up to the actors model post, below. That was a fairly generic advocation and this is a specific demonstration of how to build a small part of such a system.

In an actors model (in mine at least) actors only ever block on message receive. Blocking on I/O is not an option and so one uses all the usual techniques of setting O_NONBLOCK. One actor in the system is special, however, and blocks on an I/O multiplexing call (select, poll etc). This actor sends signals to other actors when interesting I/O is ready.

So assume that the I/O actor is using select/poll to wait for events. Data arrives from the network, sending the descriptor high, and the poll call returns. The I/O actor fires a message off to the correct actor and carries on.

However, the next poll call will return immediately because the other actor probably hasn't had a chance to perform a read and empty the kernel buffers yet. So the only option is to remove the descriptor from the set of `interesting' ones and force any actor which wants to do I/O to reenable it with a message as soon as they have finished reading.

This is a mess as it involves lots of messages going back for forth. There is a better way.

Recent multiplexing calls (epoll under Linux 2.6, kqueue under FreeBSD and realtime signals under Linux 2.4) have an edge triggered mode. In this mode the call only delivers events on a rising edge. So if a socket becomes readable it will tell you once. select and poll will tell you whenever the socket is readable.

This is clearly ideal for an actors model. The I/O actor waits for edge notifications and sends messages. The descriptor doesn't need to be removed from the interesting set and another actor can read the data at its leasure.

In the case of flow control even the descriptor need not be removed. An actor can just ignore the edge message if it doesn't currently want more data. In the future it can read until EAGAIN and then start waiting for edge messages again.

Thus my actors which talk to a socket end up looking like this:

def run(self):
	send-message-to-io-actor-registering-my-interest-in-a-certain-fd()

	while True:
		if interested-in-reading-data:
			read-data-until-EAGAIN()

		message = receive-message()
		if message == edge-notification-message:
			continue
		...