[Twisted-Python] Advice on Writing a Custom Twisted Stackless Reactor

Jean-Paul Calderone exarkun at divmod.com
Thu Jan 11 17:35:36 EST 2007


On Thu, 11 Jan 2007 13:46:17 -0800 (PST), Andrew Francis ><20070110233243.11447.1076809328.divmod.quotient.12503 at ohm>
>>Okay.  I see that you're writing stackless
>integration
>>using TSR.  I'm not sure what the point of this is.
>>Can you explain the functionality that's gained by
>>doing this, rather than using something like what's
>in >Chris Armstrong's sandbox in threadless.py?
>
>Thank you Jean-Paul for your response. My goal is to
>write a custom Twisted reactor for Stackless Python.
>Although I have had success writing programmes that
>use deferred (for client HTTP calls) and HTTP request
>handlers, I have had problems doing both
>simultaneously in one programme.
>
>Some context : I am prototyping WS-BPEL (Web services
>Business Process Execution Language) processors. In
>the WS-BPEL language, it is possible for some
>process's "threads" to act as a server and other
>"threads" to act as a client simultaneously.

This is possible without involving ThreadedSelectReactor
or Stackless Python, of course.  There may be a reason to
bring Stackless Python in to simplify the expression of
some ideas, but I'm still not sure I understand the reason
to bring in ThreadedSelectReactor.  This may just be due
to the fact that I haven't written a lot of code using
Stackless Python.  Some examples would be helpful here.

>
>The problem that I have found is that when Twisted
>blocks waiting for a server request, Stackless will
>also block, hence no progress on other tasklets that
>could execute.

Twisted will block while there are no events to process.
As soon as there is an event, though, it will wake up.  So,
if your tasklets are I/O bound, as soon as there is a network
event, Twisted will tell you about it and you can schedule
the next one to run.  If your tasklets are based on timing
events, as soon as the time comes for one of these to fire,
Twisted will also wake up and you can schedule one.  If there
is some other kind of event source, it may be a little more
challenging, but it sounds like that's not the case here.

>I thought I got around this problem by
>using task.LoopingCall/() that would issue a
>stackless.schedule() at regular intervals. However
>this approach turns out to be problematic. 

What problems did you face?  It certainly doesn't sound ideal,
but it does sound like it should give you something which does
more or less work.  Better would be to associate network and
timing events with specific tasklets and wake them up when the
reactor fires those events.

>I thought I
>could solve the problem by placing Twisted in a thread
>but no. The main problem with placing Twisted in a
>thread is that I seem to lose the deferred (and I
>don't believe I am getting deadlock).

Most Twisted APIs aren't threadsafe, so most likely what is
happening here is that you are using a Twisted API from the
wrong thread and it just isn't working right, which often
manifests as an event which never fires.

However, I still suspect bringing in a native thread is not
necessary to accomplish your goals.

>
>>Can you explain the functionality that's gained by
>>doing this,
>
>1. I solve my problem in a clean fashion.

Always a bonus. :)

>
>2. If I write a custom Stackless Reactor, then I have
>a solution that conforms to the way Twisted does
>stuff. I think would make it easier for others to use
>my stuff.

Possibly.  On the other hand, it may just introduce the
need for other people to be careful about threadsafety.

>
>>rather than using something like what's in Chris
>>Armstrong's sandbox in threadless.py?
>
>Chris's approach looks like the approach I currently
>use.

Unfortunately I still haven't managed to grasp why this
is undesirable. :(

>
>>Threaded select reactor actually isn't the same kind
>>of reactor as the rest of these, since it isn't
>>intended to be used via the normal reactor API (ie,
>>run).
>
>Right now I am using Twisted 2.4 with Stackless 2.4.3.
>
>For now, I would like to build on top of
>ThreadedSelectReactor.
>
>Back to my original question, given my previous
>example's "main loop", do I simply use interleave() to
>get my processor to call Twisted?

Roughly, yes.  blockingdemo.py in the examples directory
gives a fairly short, self-contained example of integrating
another loop using TSR.  If you haven't seen it yet, it'll
probably help or help verify your understanding.  If you have,
it might be a good basis to guide further questions.

Jean-Paul




More information about the Twisted-Python mailing list