[Twisted-Python] Simultaneous Blocking Operations and Twisted

[Matthew Glubb]

Hi All,

I am new to twisted and python. I am evaluating the possibility of  
using it for an application that I am writing. The framework looks  
very comprehensive (I am also enjoying the python language :) but I  
have some questions regarding deferred objects and the twisted web  
server itself.

My application essentially comprises the following:

1. HTTP request is received by the server.
2. Server creates a listening socket on an arbitrary port and waits  
for n seconds for incoming data
3. Data (or timeout message) is returned to the client by way of the  
HTTP response.

This is a relatively simple application and I have created a  
prototype in PHP with fastcgi. I am well aware of the limitations of  
PHP when it some to asynchronous events. The application needs to  
service thousands of clients simultaneously. In terms of CPU power,  
this is not a problem, the logic is simple and there is little  
overhead having thousands of listening sockets waiting for data (I'd  
prefer to use epoll but select or poll will do). However, there is a  
massive memory overhead because it is necessary to ensure that there  
is always one, idle process for any incoming request. Given that the  
request may wait for up to n seconds before returning a response, the  
number of simultaneous PHP fastcgi processes that are required is  
unreasonably high.

I have toyed with the idea of writing a small (low memory) binary  
that will accommodate my needs, but my C isn't great, hence twisted.

I was wondering whether it is feasible to use the twisted framework  
for this application. I have been playing with the framework and I  
can't see a way to service what are essentially thousands of requests  
that perform blocking operations simultaneously. I have noted the  
twisted documentation on threads but I was hoping, for the sake of my  
sanity, that I might be able to manage this using an asynchronous  
approach. Essentially, what I think I am looking for is a waitFor()  
on a deferred callback.

As an aside, is it possible to compile platform specific python  
applications as small binaries? Or does the interpreter (is it  
required?) mean the memory consumption is prohibitive?

Can anybody offer any advice on the matter?

Thanks a lot :)


Matthew Glubb