[Twisted-Python] integrating CompStrm//adding background processing

[Bill la Forge]

If it looks like polling and acts like polling, when is it not polling?
--When its incrimental computing!
 
I've updated the project web pages to cover the Twisted integration:
    http://compstrm.sourceforge.net
But It really does not address this issue.
 
Polling, or rather, unnecessary polling is EVIL, especially in a 
framework like Twisted. Its an even greater sin than using BLOCKING
I/O for reading email files that could be very very large. ;-)
 
But polling is when you are checking something, like I/O completion.
That's NOT what is happening in CompStrm. Instead, CompStrm is
executing (hopefully!) useful application code, in small chunks.
 
When is this useful? Well, when you have some long-running 
application code, the alternative is to execute it on a separate thread.
Indeed, this is a good solution. Except that not everyone can debug
some of the nasties that occur. And then you've got all that overhead
and the delays that seem to be part of inter-task communication.
So you start optimizing to pass large chuncks between threads and,
woops, there goes your response time. (The old fast or cheap choice.)
 
CompStrm also integrates well with async I/O, allowing you to untwist
your logic a bit. Have you read anything about Stackless Python? This
is similar, but builds on Python Generators instead.
 
Does any of this help?
 
And as for applications, indeed, CompStrm developed from the latest 
requests my client has given me. I've got a client/server app with an
asynchronous interface used to compute just-in-time displays of multiple
streams. Now he wants the existing operations pushed into scripts.
With compstrm, I can have light-weight child processes sharing the
same I/O streams. I want to port all this to Twisted, but need several
additions: CompStrm and a good bsddb integration.
 
Now that I've completed the Twisted/CompStrm integration, I'll turn to
Twisted/bsddb integration. Be assured, I will not include CompStrm in
that integration!!! (It should be generally useful, and provide a means
of reading potentially large data using server threads, while minimizing
inter-thread overheads.)
 
Bill

Christopher Armstrong <radix at twistedmatrix.com> wrote:
Indeed, it sounds like what you're doing doesn't make any sense. Why do 
you need to call something every mainloop iteration? That's exactly the 
worst way to write asynchronous code. Polling sucks. :)

Maybe you can explain more about what you're doing? What *applications* 
do you have in mind that this thing would be good for? (I notice that 
your web site's first sentence mentions how great it would be to create 
applications from components, but the web site doesn't mention anything 
at all about existing or even theoretical applications).

Yahoo! India Matrimony: Find your partner online.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20040517/9e113290/attachment.html>