[Twisted-Python] Losing Deferreds : Re: More Questions about Custom Reactors

Andrew Francis andrewfr_ice at yahoo.com
Thu Jan 25 11:44:01 EST 2007


Message: 3
Date: Mon, 22 Jan 2007 00:32:16 +0000
From: Phil Mayers <p.mayers at imperial.ac.uk>
Subject: Re: [Twisted-Python] More Questions about
Custom Reactors
To: Twisted general discussion
<twisted-python at twistedmatrix.com>
Message-ID: <45B40610.7020607 at imperial.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1;
format=flowed

I am sorry that I am tardy in responding. I was trying
out your suggestions. Right now, my main question is
under which conditions do deferred fail to fire? How
do I know that the reactor is still running? Would
Manhole help? (Unfortunately I haven't been able to
get gtk to work Manhole with Twisted 2.4)

>Perhaps you could explain what you're actually trying
>to achieve, rather than the solution you have decided
>on? An indeed, why you want to use stackless at all -
>Twisted is predicated around the assumption of a 
>stackful i.e. single-threaded Python.

I understand your questioning. The main reason I am
using Stackless Python is for its process execution
state pickling abilities. A second reason is that I
find Stackless's communication model very handy for
modeling the type of thread synchronisation I wish to
do.

I know that Twisted assumes a stackful Python and it
is an uphill battle. However the gains from getting
Twisted to work with Stackless are so great, I think
it is worthwhile. For the most part, I have succeeded
in having Stackless Python work with Twisted. If you
read the Stackless mailing list, I have posted
examples. Soon, I will post on the Wiki.

My problems stem from situations where I have two or
more concurrent tasklets (Stackless threads) that do
the following)

tasklet[0] - waiting for a Twisted Request Handler
tasklet[1] - making a client.getPage call
tasklet[2] - making a client.getPage call
tasklet[3] - some other operation
tasklet[n] - some other operation

under these conditions, if tasklet[0] is executed
first, the other tasklets will block indefinitely. My
problems occur from trying to get around this.
Originally I tried taskLoop calling
stackless.schedule() to give other tasklets a chance
to execute but this would cause reactor problems.
Hence the current approach of looking at getting
Twisted to run in its own thread.

>Alternatively you might be wanting long-running jobs
>to be interleaved with other code transparently but
>without using real threads?

I am not sure how to do this.

>If you're proposing using queues to communicate
>between a Twisted thread running the reactor and a
>stackless thread running your micro-threads, 
>I don't see why you need to modify Twisted at all. 

I am a newbie to Twisted. Based on what I am reading,
I current assumption is that I would need to 

>You can just run the stackless microthreads from one
>"real" thread (started via callInThread), pass jobs
>from Twisted to the "real" thread and probably 
>a stackless uthread, and pass the replies back to the
>twisted thread with a callFromThread.

>So, can you not just do this?:

Philip, first the good news : thanks a million! This
is the exactly the type of example I wanted. It works.

The bad news - under certain conditions I am losing
deferreds. I think there are deadlock issues but that
is not Twisted's fault per se. Unfortunately debuggers
work poorly with Stackless so it is difficult for me
to know what is happening. I tried telneting but my
server is not responding - so I guess that implies
that Twisted is still listening but the reactor is not
working. Again, any suggestions would be more than
welcome.

Cheers,
Andrew


 
____________________________________________________________________________________
Bored stiff? Loosen up... 
Download and play hundreds of games for free on Yahoo! Games.
http://games.yahoo.com/games/front




More information about the Twisted-Python mailing list