[Twisted-Python] Two main loops

Jean-Paul Calderone exarkun at divmod.com
Mon Nov 12 13:15:47 MST 2007


On Mon, 12 Nov 2007 20:08:29 +0100, Nitro <nitro at dr-code.org> wrote:
>Am 12.11.2007, 19:47 Uhr, schrieb Jasper <jasper at peak.org>:
>>To get to the point, I actually do feel that the fault lies with  Twisted, 
>>or more specifically PB.  Code using PB wants to be as unaware  of the 
>>details of network transmission as possible.  It certainly  doesn't want to 
>>slice things into a bazillion chunks for individual  transmission, and then 
>>puzzle them back together on the other end.
>>
>>IMHO it makes more sense for PB to understand it shouldn't block, and to 
>>chunk and schedule its serialization (and unserialization) in the same 
>>manner as the TCP packets sent underneath.  I'm guessing this would make 
>>for cleaner separation, and more general usefulness.
>
>I think this is a) not possible and b) not desirable.

I think it's possible.  It may even be desirable, although whether it is
worth the cost of implementing or not is a question worth considering.

>It's not possible 
>because pb sends a single large chunk of data over tcp.

It doesn't have to be implemented this way.  However, this isn't
particularly relevant.  ITCPTransport.write takes approximately the same
amount of time to execute and return whether you pass it a string which is
one byte long or one million bytes long.

>The OS is 
>responsible to break everything down into individual tcp packets and  puzzle 
>them together at the other end. The whole packet thing is not  visible to 
>pb.

True, but the actual performance bottleneck here is in PB's conversion of
Python objects to strings, which is done by code entirely within Twisted.

>So if you want pb to split your data, then do splitted  callRemotes.
>It's not desirable because you don't want your program to be stalled at 
>uncontrollable times when the OS decides to send 20 tcp packets at once. 

Your program won't ever stall because of this.

>You want to spread the load as evenly as possible and you'll have to do 
>this manually.

PB's serializer could try very hard to avoid running for a long period of
time without giving control back to the reactor.  Of course, someone would
have to implement this.  Whether or not that is worthwhile to implement,
instead of doing what Glyph suggested (manually breaking up the work into
smaller pieces) is a separate question.

Jean-Paul




More information about the Twisted-Python mailing list