[Twisted-Python] Flow-control for Pipes / Unix Domains Sockets?

[Tobias Oberstein]

Hi Itamar,

once again Twisted surprises me: it just works!

I have tested a master/child combo over pipe (both Twisted) with a push producer on the sending leg.

It can push around 70MB/s raw binary (16k write size) to the child. Load is evenly on 2 CPU cores each at 80% and stays roughly constant. Memory is flat for both.

This is on some years old Macbook (kqueue reactor .. a little faster than select) - tomorrow I repeat on a beefy system/OS.

I will do more systematic throughput and also latency measurements .. but this is already not bad at all.

Moreso: I really am curious now how that works under the hood, since the pausing/resuming seems to be upper/lower watermark controlled .. it'll resume the producer before the consumer starves. As it should be;)

Background: this is part of experiments in preparation for a multi-core capable Autobahn based message broker ..

Takeaway: Next time I don't waste time on the internet reading half-baked posts, but just hack away;)

/Tobias

PS: The following is actually slower than above master/slave pair .. which I also didn't expect:

tobias-obersteins-macbook-pro-2:masterchild oberstet$ time dd if=/dev/zero bs=1k count=1000000 | wc -c
1000000+0 records in
1000000+0 records out
1024000000 bytes transferred in 17.454178 secs (58667902 bytes/sec)
 1024000000

real    0m17.460s
user    0m18.389s
sys    0m7.729s



On 22.10.13 02:00, "Itamar Turner-Trauring" <itamar at itamarst.org> wrote:


On 10/21/2013 04:48 PM, Tobias Oberstein wrote:


Flow-control for Pipes / Unix Domains Sockets? Hi,

 with TCP (either remote or loopback) I can have flow-control using the producer-consumer machinery that Twisted provides.

 Is that (flow-control / producer-consumer) also available (and practically usable/recommended) with:

 a) Unix Domain sockets
 b) Pipes (spawnProcess)


 I would expect transport.registerProducer to work with both.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20131022/d39cb099/attachment.html>