[Twisted-Python] Re: Advice on Writing a Custom Twisted Stackless Reactor

Andrew Francis andrewfr_ice at yahoo.com
Fri Jan 12 16:21:45 EST 2007

Hello Jean-Paul:

Once again, thanks for your response. 

>Twisted will block while there are no events to
>process. As soon as there is an event, though, it
will >wake up.  So, if your tasklets are I/O bound, as
soon >as there is a network event, Twisted will tell
you >about it and you can schedule the next one to

I know this. Since Twisted and Stackless are using the
same OS thread, this will block Stackless. Stackless
will block under the following condition - 

some notation:

S - tasklet waiting for a server call (waiting for a
Twisted request Handler )
C - tasklet waiting for a client call (deferred
| - running in parallel 

if you have a sequence of the form (S | C[0] | C[1] |
c[n]) and S executes first, there will be no progress
on the client tasklets. I tried adding a timer but had
limited success.

(By the way, when I first mentioned this in the
Stackless mailing list, Bob Ipolito pointed this out
to me)

AF>I thought I got around this problem by
AF>using task.LoopingCall/() that would issue a
AF>stackless.schedule() at regular intervals. However
AF>this approach turns out to be problematic. 

>What problems did you face?  It certainly doesn't
>sound ideal, but it does sound like it should give
you >something which does more or less work.

In some tests I would get a Twisted error :

why = getAttr(selectable, method)()

others times my processor would simply hang because
the deferred function was not called.

from my logging, the common demoninator was that the
task.LoopingCall stopped.

>Better would be to associate network and
>timing events with specific tasklets and wake them up
>when the reactor fires those events.

I do this. 

>Most Twisted APIs aren't threadsafe, so most likely
>what is happening here is that you are using a
Twisted >API from the wrong thread and it just isn't
working >right, which often manifests as an event
which never >fires.

What is the wrong thread? How do I correct this?

>However, I still suspect bringing in a native thread
>is not necessary to accomplish your goals.

Maybe you are right. So far my experiences are
pointing in the opposite direction. Also I have read
posts involving integrating Chandler and IronPython
with Twisted and they are running Twisted in a
separate thread. 

That said, these are my deferred calls :

postRequest = client.getPage(self.address, method =
'POST', headers = self.headers, postdata = self.body) 

How should these be done from a thread?

>Roughly, yes.  blockingdemo.py in the examples
>directory gives a fairly short, self-contained
example >of integrating another loop using TSR.  If
you haven't >seen it yet, it'll probably help or help
verify your >understanding.  If you have, it might be
a good basis >to guide further questions.

I looked at blockingdemo.py. Blockingdemo is
structured different from the reactor examples in the
Twisted.internet module. Most of the example reactors
i have seen inherit from threadselectreactor and
override the run() method. What does
TwistedManager.poll() do? The method is defined but
called nowhere. I commented out poll() and the
programme did the same thing. 

That said, I have been looking at the wxreactor.py. 

My questions are:

1. With the customer reactor, it would be Stackless
that drives Twisted, not the other way around?

2. How exactly does interleave work? Some reactors
don't use it. My Stackless thread and Twisted talk
through stackless channels and a variation of a
stackless channel, the uthread Queue.

My strategy is as follows:

1. Inherit from threadselectreactor and override run()

2. in run(), my interleave calls __hook__()

3. In my Stackless Application, write a method 
__hook__. __hook__ looks like wxCallAfter in wxPython

def __hook__(self, function, *args, **kw):
    return function(*args, **kw)

3. Since stackless.schedule() is often called to yield
control, wrap the stackless.schedule() 

call the underlying Stackless scheduler
def __schedule__(self):
    # this is where I am a little fuzzy but
    # has to be called somehow
4. In the StacklessReactor.run(), the main loop that
is called will look like:
def MainLoop(self):

    while(stackless.getruncount() > 1):

Again, I really appreciate insights. I really want to
get Stackless Python to properly work with Twisted.


8:00? 8:25? 8:40? Find a flick in no time 
with the Yahoo! Search movie showtime shortcut.

More information about the Twisted-Python mailing list