[Twisted-Python] CONNECTION_LOST not an integer (docstring error?)

[Michael McNeil Forbes]

> On Oct 1, 2008, at 3:44 PM, Jean-Paul Calderone wrote:
...
>> Why does writeSomeData not simply raise CONNECTION_LOST as an   
>> exception? Checking return values is quite un-pythonic.  Is there  
>> a  deep reason for this?
>
> The reason is probably that whoever wrote it thought that raising an
> exception would be enough slower than returning a special value that
> it would be worth doing something slightly unusual.  Compared to
> returning a value, raising an exception is expensive in CPython, but
> I doubt there is any surviving benchmark which demonstrates that it
> makes a difference here.
>
> However, it's a very low-level API and since it isn't intended to be
> used by application code, it doesn't seem like a very high priority
> to make it less confusing.

By the way, is there any convention by which "high level" API  
routines are distinguished from "low level" routines?  It is really  
not that clear from the docs... (writeSomeData comes first in the  
listing etc.)

>> P.S. I came across this because I was trying to use twisted  
>> running  in a thread to write data resulting from a long  
>> computation that I  have not yet turned into a producer.  The more  
>> conventional "write"  method was failing if the socket backed up,  
>> and provided no simple  way of determining if data was being  
>> dropped.  My solution was to  used writeSomeData, and then have  
>> the computation decide to throw out  some of the data if it is  
>> being produced too rapidly, but I need to  know how much has been  
>> sent so I can decide what to throw away...
>>
>
> This seems more interesting than the main point of your email. :) You
> should probably just do the work to turn the computation into a  
> producer.
> This because:
>
>  * writeSomeData is low level and crufty, as you've discovered
>  * Not all reactors have this API, so your program will break based on
>    the reactor selected to run it
>  * producers give you all the same information
>
> Since you mentioned threads, I'll also point out that Twisted APIs  
> are not
> safe to use from non-reactor threads without  
> reactor.callFromThread.  If
> you aren't using callFromThread, that would explain bad behavior  
> from the
> write method.

I know that and am using callFromThread: The reason I am having bad  
behavior is because the socket is backing up.  Once I put in the  
logic to throw out data, it works fine... but the lack of reactor  
support concerns me somewhat.  I don't want the computation to stop,  
but think I could put an intermediate "push producer" that stops by  
discarding the data.

I am threading because the computation is the main task, and the  
network stuff is simply a window into the state of the computation.   
I want the computation to run continually, and it needs to do so in a  
blocking sort of way, periodically checking and pushing the partial  
results (like once every 30 minutes or so).

I want the calculation code to function on its own, with or without  
the twisted components and I don't see a simple way of doing this if  
twisted is in complete control without some type of threading.

My current design is to write my calculation as if twisted did not  
exist, and just push the current results onto a Queue periodically.   
Then I run twisted in a separate thread with clients that  
periodically check the queue, pop stuff off and dispatch it to any  
clients who happen to connect and want the data.

Is there a simpler way to do this all with twisted without using  
threads?  (The main goal is having an extremely minimal set of hooks  
for the people writing the computational code, and having the  
computational code run as fast as possible.)

Thanks for all of the suggestions and prompt comments!  (And for one  
very "twisted" library;-)
Michael.