[Twisted-Python] Performance issues of twisted.

[glyph at divmod.com]

On 13 Apr, 04:33 pm, newptcai at gmail.com wrote:
>Hi all!
>
>I am preparing to write a server application (you may think of it as an
>online game server) with twisted, but I am worrying about it's 
>performance
>and how to do it correctly.

If this is the first time you have done such a thing, let me give you a 
friendly piece of advice: think about how to do it correctly first, then 
think about how to speed it up once it works.  This is common wisdom in 
programming, but it is especially important when you think you have 
found an exception to the rule, and stuff needs to go *really fast*, 
*right away*.  If you try to make it perform well before understanding 
what's going on, it will be both slow *and* full of mistakes and 
complexity from misguided implementations.

Ideally, you could work on an existing project, such as MV3d ( 
http://www.mv3d.com/trac ) rather than starting from scratch.
>For example..
>
>1. UDP or TCPÿ0Cwhich should I choose to gain higher performance?

If you do not already know the answer to this question, choosing UDP is 
very dangerous.  For what TCP does, TCP is higher performance, and a lot 
less buggy, than anything you would likely implement to replace it.

In a few *very specialized* applications, in specific network 
conditions, UDP can provide better performance.  However, performance 
does not usually mean "speed" in those circumstances; it means the 
ability, for example, to drop a bunch of packets and then pick up again 
without catching up to the packets that were lost in the middle.  But 
picking up packets that were lost in the middle is a very useful 
feature, one that most applications rely on without even knowing it.
>2. Should I catch data in memory instead of write it to db immediately?

This question could mean almost anything.  However, writing to the 
database is simpler and less error-prone, so start there, and *only if 
the code is not fast enough*, add a caching optimization.  Caching is 
generally more useful on reads than writes, though, so I don't really 
understand your question.
>3. Could I make it run on cluster if cache data in memory  15 15 I think it
>might be quit hard to exchange data between instances of the server if 
>I
>cache data in memory.

This is why you should
>4. How to make hot backup?

http://www.oracle.com/technology/documentation/berkeley- 
db/db/ref/transapp/archival.html

(This is not a helpful answer, but hints at the complexity of a real 
answer to this question, depending on the specifics of what database 
you're using.  It's not something for a one-liner in an email.)
>5. Garbage collection might make the server halt for a moment

Yep.  Of course, manual memory allocation might make the server halt 
forever, because it segfaulted.  Take your pick.
>6. What is happening in a computer when an IP package received?

Start here, and click on every link:

http://en.wikipedia.org/wiki/Internet_protocol_suite
>7. Could I get some inspiration from how people write web server?

Start here, and click on every link:

http://twistedmatrix.com/trac/browser/trunk/twisted/web/
>8. If i use an separate physical server to deploy the database, could I 
>gain
>some performance improvement?  Or the cost of communication between the 
>db
>and the server could hurt the total performance?

Maybe.  The way that you tell is not by asking a question on a mailing 
list, but by implementing a specific strategy and then measuring it to 
see what its bottlenecks are and where you can improve performance.  If 
your experience is anything like mine, you will discover something 
surprising.  For example, that everything except the server-side 
collision detection is so fast that you don't need to bother to optimize 
it, but the collision detection is so slow that you need to change it 
radically to even have the simulation keep up with wall clock time.
>Also, is there any classical books talking about these kind of issues?

I wrote an article in one once:

http://tinyurl.com/4yy8zx

but this book, nor any other, won't answer all your questions.  The 
field, and the industry, is way too new.