[Twisted-Python] fast high load protocol
phil at bubblehouse.org
Tue Feb 16 15:08:46 EST 2010
On Feb 16, 2010, at 2:46 PM, Vlad Shevchenko wrote:
> I have developed two kind of servers:
> api - http server
> memory_db - dict-based server with some logic of putting data to dict and getting it back.
> Api communicate with memory_db via txJSON-RPC but response time is huge under high load. I also try implementation with replacement txJSON-RPC -> Perspective Broker with no success. PB implementation has 10+ times faster responses when returning data is small (dict with few keys and values like list of 1-5 strings). However, PM much depends on size of returning value. When I return a list of 600 items (each is string with length 20) response time incredibly growing up. Unlike PB, json-rpc implementation keeps it response-time near the same lever as with small return data.
> Can you suggest some protocol with a small response? Or should I look into development of my own?
It definitely depends on your use case...
PB can still be a good solution for you if you change your architecture a little bit. Instead of returning a 600-item list, you could return a pb.Referencable object that can iteratively return each row at a time.
You may also want to look into using Copyable or Cacheable PB objects. More on that here:
Of course, if you need a large amount of data to be available before you can begin calculations or whatnot, you're going to have to wait for it to be sent no matter what protocol you use.
More information about the Twisted-Python