[Twisted-Python] Simultaneous Blocking Operations and Twisted
matt at zgroupplc.com
Fri Mar 30 07:01:27 EDT 2007
I am new to twisted and python. I am evaluating the possibility of
using it for an application that I am writing. The framework looks
very comprehensive (I am also enjoying the python language :) but I
have some questions regarding deferred objects and the twisted web
My application essentially comprises the following:
1. HTTP request is received by the server.
2. Server creates a listening socket on an arbitrary port and waits
for n seconds for incoming data
3. Data (or timeout message) is returned to the client by way of the
This is a relatively simple application and I have created a
prototype in PHP with fastcgi. I am well aware of the limitations of
PHP when it some to asynchronous events. The application needs to
service thousands of clients simultaneously. In terms of CPU power,
this is not a problem, the logic is simple and there is little
overhead having thousands of listening sockets waiting for data (I'd
prefer to use epoll but select or poll will do). However, there is a
massive memory overhead because it is necessary to ensure that there
is always one, idle process for any incoming request. Given that the
request may wait for up to n seconds before returning a response, the
number of simultaneous PHP fastcgi processes that are required is
I have toyed with the idea of writing a small (low memory) binary
that will accommodate my needs, but my C isn't great, hence twisted.
I was wondering whether it is feasible to use the twisted framework
for this application. I have been playing with the framework and I
can't see a way to service what are essentially thousands of requests
that perform blocking operations simultaneously. I have noted the
twisted documentation on threads but I was hoping, for the sake of my
sanity, that I might be able to manage this using an asynchronous
approach. Essentially, what I think I am looking for is a waitFor()
on a deferred callback.
As an aside, is it possible to compile platform specific python
applications as small binaries? Or does the interpreter (is it
required?) mean the memory consumption is prohibitive?
Can anybody offer any advice on the matter?
Thanks a lot :)
More information about the Twisted-Python