[Twisted-Python] CONNECTION_LOST not an integer (docstring error?)
Michael McNeil Forbes
mforbes at physics.ubc.ca
Wed Oct 1 18:37:47 EDT 2008
> On Oct 1, 2008, at 3:44 PM, Jean-Paul Calderone wrote:
>> Why does writeSomeData not simply raise CONNECTION_LOST as an
>> exception? Checking return values is quite un-pythonic. Is there
>> a deep reason for this?
> The reason is probably that whoever wrote it thought that raising an
> exception would be enough slower than returning a special value that
> it would be worth doing something slightly unusual. Compared to
> returning a value, raising an exception is expensive in CPython, but
> I doubt there is any surviving benchmark which demonstrates that it
> makes a difference here.
> However, it's a very low-level API and since it isn't intended to be
> used by application code, it doesn't seem like a very high priority
> to make it less confusing.
By the way, is there any convention by which "high level" API
routines are distinguished from "low level" routines? It is really
not that clear from the docs... (writeSomeData comes first in the
>> P.S. I came across this because I was trying to use twisted
>> running in a thread to write data resulting from a long
>> computation that I have not yet turned into a producer. The more
>> conventional "write" method was failing if the socket backed up,
>> and provided no simple way of determining if data was being
>> dropped. My solution was to used writeSomeData, and then have
>> the computation decide to throw out some of the data if it is
>> being produced too rapidly, but I need to know how much has been
>> sent so I can decide what to throw away...
> This seems more interesting than the main point of your email. :) You
> should probably just do the work to turn the computation into a
> This because:
> * writeSomeData is low level and crufty, as you've discovered
> * Not all reactors have this API, so your program will break based on
> the reactor selected to run it
> * producers give you all the same information
> Since you mentioned threads, I'll also point out that Twisted APIs
> are not
> safe to use from non-reactor threads without
> reactor.callFromThread. If
> you aren't using callFromThread, that would explain bad behavior
> from the
> write method.
I know that and am using callFromThread: The reason I am having bad
behavior is because the socket is backing up. Once I put in the
logic to throw out data, it works fine... but the lack of reactor
support concerns me somewhat. I don't want the computation to stop,
but think I could put an intermediate "push producer" that stops by
discarding the data.
I am threading because the computation is the main task, and the
network stuff is simply a window into the state of the computation.
I want the computation to run continually, and it needs to do so in a
blocking sort of way, periodically checking and pushing the partial
results (like once every 30 minutes or so).
I want the calculation code to function on its own, with or without
the twisted components and I don't see a simple way of doing this if
twisted is in complete control without some type of threading.
My current design is to write my calculation as if twisted did not
exist, and just push the current results onto a Queue periodically.
Then I run twisted in a separate thread with clients that
periodically check the queue, pop stuff off and dispatch it to any
clients who happen to connect and want the data.
Is there a simpler way to do this all with twisted without using
threads? (The main goal is having an extremely minimal set of hooks
for the people writing the computational code, and having the
computational code run as fast as possible.)
Thanks for all of the suggestions and prompt comments! (And for one
very "twisted" library;-)
More information about the Twisted-Python