[Twisted-Python] Losing Deferreds : Re: More Questions about Custom Reactors

Andrew Francis andrewfr_ice at yahoo.com
Thu Jan 25 11:44:01 EST 2007

Message: 3
Date: Mon, 22 Jan 2007 00:32:16 +0000
From: Phil Mayers <p.mayers at imperial.ac.uk>
Subject: Re: [Twisted-Python] More Questions about
Custom Reactors
To: Twisted general discussion
<twisted-python at twistedmatrix.com>
Message-ID: <45B40610.7020607 at imperial.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1;

I am sorry that I am tardy in responding. I was trying
out your suggestions. Right now, my main question is
under which conditions do deferred fail to fire? How
do I know that the reactor is still running? Would
Manhole help? (Unfortunately I haven't been able to
get gtk to work Manhole with Twisted 2.4)

>Perhaps you could explain what you're actually trying
>to achieve, rather than the solution you have decided
>on? An indeed, why you want to use stackless at all -
>Twisted is predicated around the assumption of a 
>stackful i.e. single-threaded Python.

I understand your questioning. The main reason I am
using Stackless Python is for its process execution
state pickling abilities. A second reason is that I
find Stackless's communication model very handy for
modeling the type of thread synchronisation I wish to

I know that Twisted assumes a stackful Python and it
is an uphill battle. However the gains from getting
Twisted to work with Stackless are so great, I think
it is worthwhile. For the most part, I have succeeded
in having Stackless Python work with Twisted. If you
read the Stackless mailing list, I have posted
examples. Soon, I will post on the Wiki.

My problems stem from situations where I have two or
more concurrent tasklets (Stackless threads) that do
the following)

tasklet[0] - waiting for a Twisted Request Handler
tasklet[1] - making a client.getPage call
tasklet[2] - making a client.getPage call
tasklet[3] - some other operation
tasklet[n] - some other operation

under these conditions, if tasklet[0] is executed
first, the other tasklets will block indefinitely. My
problems occur from trying to get around this.
Originally I tried taskLoop calling
stackless.schedule() to give other tasklets a chance
to execute but this would cause reactor problems.
Hence the current approach of looking at getting
Twisted to run in its own thread.

>Alternatively you might be wanting long-running jobs
>to be interleaved with other code transparently but
>without using real threads?

I am not sure how to do this.

>If you're proposing using queues to communicate
>between a Twisted thread running the reactor and a
>stackless thread running your micro-threads, 
>I don't see why you need to modify Twisted at all. 

I am a newbie to Twisted. Based on what I am reading,
I current assumption is that I would need to 

>You can just run the stackless microthreads from one
>"real" thread (started via callInThread), pass jobs
>from Twisted to the "real" thread and probably 
>a stackless uthread, and pass the replies back to the
>twisted thread with a callFromThread.

>So, can you not just do this?:

Philip, first the good news : thanks a million! This
is the exactly the type of example I wanted. It works.

The bad news - under certain conditions I am losing
deferreds. I think there are deadlock issues but that
is not Twisted's fault per se. Unfortunately debuggers
work poorly with Stackless so it is difficult for me
to know what is happening. I tried telneting but my
server is not responding - so I guess that implies
that Twisted is still listening but the reactor is not
working. Again, any suggestions would be more than


Bored stiff? Loosen up... 
Download and play hundreds of games for free on Yahoo! Games.

More information about the Twisted-Python mailing list