<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><title>Re: [Twisted-Python] Flow-control for Pipes / Unix Domains Sockets?</title><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Vorformatiert Zchn";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.E-MailFormatvorlage17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.HTMLVorformatiertZchn
{mso-style-name:"HTML Vorformatiert Zchn";
mso-style-priority:99;
mso-style-link:"HTML Vorformatiert";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=DE link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Ok, I've pimped up the test<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><a href="https://github.com/oberstet/scratchbox/blob/master/python/twisted/masterchild/streaming_master.py">https://github.com/oberstet/scratchbox/blob/master/python/twisted/masterchild/streaming_master.py</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><a href="https://github.com/oberstet/scratchbox/blob/master/python/twisted/masterchild/streaming_child.py">https://github.com/oberstet/scratchbox/blob/master/python/twisted/masterchild/streaming_child.py</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>and running on Ubuntu LTS 12.04, 64 Bit, Core i7 3.4GHz, epoll reactor, I was able to get it up to<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Master => Child : 450MB/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>[94%/100% CPU load)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Master <=> Child (full-duplex echo): 200MB/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>[95%/100% CPU load)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Nifty.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Now, it might be that I can push even higher by adjusting the pipe buffer sizes (there is a Linux fcntl for that .. </span><b>F_SETPIPE_SZ</b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>).</span><o:p></o:p></pre><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Q: How do I get at the FDs of the underlying pipes to the spawned process in the parent?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>reactor.spawnProcess returns an object implementing </span>IProcessTransport<span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>However, that does not seem to provide access to the underlying pipes (and their FDs in the parent)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Looking at<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><a href="http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.1.0/twisted/internet/process.py">http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.1.0/twisted/internet/process.py</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>I can get it, but not via official API, right?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>===<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Windows: It does not work at all. Tested "select" and "iocp" reactors. It just hangs doing nothing.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Von:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] <b>Im Auftrag von </b>Tobias Oberstein<br><b>Gesendet:</b> Mittwoch, 23. Oktober 2013 00:58<br><b>An:</b> Twisted discussion<br><b>Betreff:</b> Re: [Twisted-Python] Flow-control for Pipes / Unix Domains Sockets?<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Hi Itamar,<br><br>once again Twisted surprises me: it just works!<br><br>I have tested a master/child combo over pipe (both Twisted) with a push producer on the sending leg.<br><br>It can push around 70MB/s raw binary (16k write size) to the child. Load is evenly on 2 CPU cores each at 80% and stays roughly constant. Memory is flat for both.<br><br>This is on some years old Macbook (kqueue reactor .. a little faster than select) - tomorrow I repeat on a beefy system/OS.<br><br>I will do more systematic throughput and also latency measurements .. but this is already not bad at all.<br><br>Moreso: I really am curious now how that works under the hood, since the pausing/resuming seems to be upper/lower watermark controlled .. it'll resume the producer before the consumer starves. As it should be;)<br><br>Background: this is part of experiments in preparation for a multi-core capable Autobahn based message broker ..<br><br>Takeaway: Next time I don't waste time on the internet reading half-baked posts, but just hack away;)<br><br>/Tobias <br><br>PS: The following is actually slower than above master/slave pair .. which I also didn't expect:<br><br>tobias-obersteins-macbook-pro-2:masterchild oberstet$ time dd if=/dev/zero bs=1k count=1000000 | wc -c<br>1000000+0 records in<br>1000000+0 records out<br>1024000000 bytes transferred in 17.454178 secs (58667902 bytes/sec)<br> 1024000000<br><br>real 0m17.460s<br>user 0m18.389s<br>sys 0m7.729s<br><br><br><br>On 22.10.13 02:00, "Itamar Turner-Trauring" <<a href="itamar@itamarst.org">itamar@itamarst.org</a>> wrote:</span><o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> <br>On 10/21/2013 04:48 PM, Tobias Oberstein wrote:<br> <br> </span><o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Flow-control for Pipes / Unix Domains Sockets? Hi,<br> <br> with TCP (either remote or loopback) I can have flow-control using the producer-consumer machinery that Twisted provides.<br> <br> Is that (flow-control / producer-consumer) also available (and practically usable/recommended) with:<br> <br> a) Unix Domain sockets<br> b) Pipes (spawnProcess)<br> </span><o:p></o:p></p></blockquote><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br> I would expect transport.registerProducer to work with both.<br> </span><o:p></o:p></p></blockquote></div></div></body></html>