[Twisted-Python] Twisted scalability question

[Phillip J. Eby]

At 11:58 PM 9/22/03 +0300, Matti Jagula wrote:

>So, after the rather lengthy introduction, I'd like to know what am I doing
>wrong? How can I show the better scalability of Twisted platform?

This is just a guess, but you probably aren't doing anything complicated 
enough to benefit.  :)

Seriously, if all you're doing is sending a single datagram in response to 
a simple DB query, framework overhead will continue to dominate your 
processing time.  Think about the two programs: both perform the query and 
send the datagram.  But the non-Twisted program *does nothing else*.  So, 
what you're seeing is that the per-invocation framework overhead for the 
Twisted version is about 3x what's required for just the db query and 
datagram send.  For something so simple, that actually sounds pretty 
reasonable!

What Twisted (and the "reactor" pattern in general) excels at is 
multiplexing *concurrent and ongoing* I/O operations.  However, your 
application has *no* ongoing I/O!  In general, Twisted scales well because 
it lets you do other things while you're waiting for I/O.  With TCP 
streams, you may have to wait before you write data, because the receiver 
may be slow.  Likewise, if you're receiving data, the sender may be 
slow.  While it's waiting, Twisted can let you do something else, like send 
or receive data from someone else that's ready at that moment, or do some 
actual computation.

However, in your setup, the *only* I/O that you ever wait for is the 
database!  You're receiving a single UDP packet (that you don't "wait" 
for), and sending a UDP packet (that you don't need to wait to 
send!).  Even if you made it take 10 times longer for the DB query to run, 
you'd *still* be able to send that data out as soon as you receive it.

In short, the problem you're solving is so darn simple that there's no way 
using Twisted can really improve on it, except in terms of issues like 
keeping your DB connection open.  There's simply nothing here for Twisted 
to multiplex!