[Twisted-Python] twisted and performance

[Christian Simms]

On 6/13/07, weiwei weiwei <waterinfire at hotmail.com> wrote:
>
> Hi,all!
>
>
>
>          Thanks for the python twisted,it is a great software.
>
>
>
>          I am from china. This days we use twisted to develop a
> middleware.
>
>
>
>          Requirement is:
>
> 1)       Receive:  receive 100~200 different client¡¯s message(they all
> connect to our listen socket) and insert it into database(oracle)
>
> 2)       Send: application insert message into db,then our middleware
> select
> message from database and send this message to its destination through
> socket.
>
>
>
> Our design is : reactor + adbapi.ConnectionPool
>
> 1)       reactor is used in receiveing and sending message from and to
> socket.
>
> 2)       when we received a message,we use adbapi to insert it to db
>
> 3)       we use adbapi to select message from db,and use a Deferred to
> notify us when we got a message from db,and send it to peer,after
> receiving
> the peer¡¯s acknowledge message ,we update the databse.
>
>
>
> After testing many times,we found when the connection number is less than
> 20,its performance is very good.But if the connection number is more than
> 30
> or 50, it is performance is worse and worse.
>
>
>
> DO you give me some advice about the design?
>
>
>
>
>
> Thanks in advance.
>
>
>
>
>
> Wei wei
>
> _________________________________________________________________
> Express yourself instantly with MSN Messenger! Download today it's FREE!
> http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>


If I had to guess, I would guess you're having database performance issues.
The overhead of TCP/IP networking of 50, or even 500 simultaneous
connections is nothing compared to the overhead of that many simultaneous
interactions with a database.  You can figure this out by doing some
experiments, like:

- run your test with a fake database which just uses an in-memory database
-- if still slow then problem is in your networking, if fast then problem is
in database

- if that's too difficult, another experiment would be to write some test
code to run a number of simultaneous database interactions without any
networking

Cheers,
Christian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20070614/738ee607/attachment.htm