[Twisted-Python] Advice on Writing a Custom Twisted Reactor

Jean-Paul Calderone exarkun at divmod.com
Wed Jan 10 18:32:43 EST 2007


On Wed, 10 Jan 2007 14:44:16 -0800 (PST), Andrew Francis <andrewfr_ice at yahoo.com> wrote:
>Hello Colleagues:
>
>I am a first time poster...

Welcome to the list. :)

>
>For the past few months I have been experimenting with
>using Stackless Python with Twisted. I still consider
>myself very much a Twisted newbie. Currently my
>problems involve deferreds.

Okay.  Hopefully this will be easy to straighten out:
Deferreds are quite simple.

>
>>From doing online searches, I have run across
>customized reactors for Twisted. Consequently I am
>looking at the custom reactors that come with Twisted.
>Like Stackless, many GUIs have a hard time working
>with Twisted which blocks. Based on the
>ThreadedSelectReactor, most of the reactors are for
>GUIs. The general idea is that custom reactor's event
>loop calls Twisted in its own thread. This approach
>seems like it would work for Stackless and provide a
>decent general solution. However there are a few
>places where I am not quite clear.

The split is pretty even, actually.  There's are GUI
reactors for Gtk, Gtk2, wxPython, and Qt.  There are
non-GUI reactors for select, poll, epoll, KQ, and IOCP.
There's also a Win32 event reactor, which is kind of
for GUI integration and kind of for other stuff.

Threaded select reactor actually isn't the same kind of
reactor as the rest of these, since it isn't intended
to be used via the normal reactor API (ie, run).  In
Twisted 2.5, we've renamed it to try to make this more
apparent: it's now in a "private" module which doesn't
have "reactor" in its name.  It still works in the
same way as it always did, though.

For most reactors, the behavior is actually to monitor
event sources and then dispatch received events to code
interested in them.  Whether these events are for GUI
elements or file descriptors doesn't really matter much.
Whether the monitoring is done in one thread or many is
also more or less irrelevant, as far as the external API
is concerned.  It _is_ important that all event dispatching
happens to a single thread, since this is what programs
written with Twisted expect to happen, and doing otherwise
would typically introduce bugs into these applications.

I'm not sure what "general solution" you have in mind, so
this may or may not be relevant to the problem you're
tackling.

>
>Most of my Stackless programmes have the following
>form (think of this as a MainLoop):
>
>~
>
>set up stuff
>
>while (stackless.getruncount() > 1):
>   stackless.schedule()
>
>~
>
>Since it seems my custom reactor has to call Twisted
>as often as possible. I would wrap
>stackless.schedule() (since this is often called):

All of the currently implemented Twisted reactors need
to wake up under two circumstances:

  * a network event has occurred and needs to be reacted to
  * a timing event has occurred and needs to be reacted to

The most correct integration will account for these two facts
and only wake up as often as is necessary to deal with the
relevant events.

Of course, waking up more often isn't necessarily incorrect,
but it is wasteful.  Also, if the scheduling strategy is "as
fast as possible", then the accuracy of timed event processing
typically suffers.

>
>stacklessReactor.schedule():
>    __hook__() # some method that would call Twisted
>    stackless.schedule()
>
>Where I start to get a bit confused is:
>
>1) In the Custom reactor.run() method, do I call
>self.__hook__() in the interleave() method? Is there
>anything more I have to do in the run() method? Can I
>avoid signal handler issues by passing False to
>Twisted's reactor.run() ?
>
>2) From my Stackless code, do I have to use
>self.callFromThread() to call Twisted methods that
>involve deferreds?
>

Okay.  I see that you're writing stackless integration
using TSR.  I'm not sure what the point of this is.  Can
you explain the functionality that's gained by doing this,
rather than using something like what's in Chris Armstrong's
sandbox in threadless.py?

>Again, any advice would be welcomed.

I don't think I've completely understood your goals, but I
hope this is a good starting point.

Jean-Paul




More information about the Twisted-Python mailing list