<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dirk,</div><div><br></div>Using deferred directly in your bin2intAsync() may be somewhat less efficient than some other way described in <span class="Apple-style-span" style="color: rgb(87, 87, 90); font-family: verdana, sans-serif; font-size: 15px; font-weight: bold; line-height: 18px; ">Recipe 439358: [Twisted] From blocking functions to deferred functions </span><div><font class="Apple-style-span" face="Helvetica, sans-serif"><br></font></div><div><span class="Apple-style-span" style="color: rgb(87, 87, 90); font-family: verdana, sans-serif; font-size: 15px; font-weight: bold; line-height: 18px; "><span class="Apple-style-span" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-weight: normal; line-height: normal; ">recipe (<a href="http://code.activestate.com/recipes/439358/">http://code.activestate.com/recipes/439358/</a>)</span></span></div><div><font class="Apple-style-span" face="Helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="Helvetica, sans-serif">You would get same effect (asynchronous execution) - but potentially more efficiently - by just decorating your synchronous methods as:</font></div><div><font class="Apple-style-span" face="Helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="Helvetica, sans-serif"><span class="Apple-style-span" style="font-family: verdana, sans-serif; border-collapse: collapse; color: rgb(87, 87, 90); font-size: small; line-height: 18px; -webkit-border-horizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px; "><pre style="font-size: 1em; background-color: rgb(241, 241, 241); border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; border-width: initial; border-color: initial; overflow: inherit; position: static; z-index: auto; "><span class="k" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; color: rgb(0, 112, 32); font-weight: bold; ">from</span> <span class="nn" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; color: rgb(14, 132, 181); font-weight: bold; ">twisted.internet.threads</span> <span class="k" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; color: rgb(0, 112, 32); font-weight: bold; ">import</span> <span class="n" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">deferToThread</span>
<span class="n" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">deferred</span> <span class="o" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; color: rgb(102, 102, 102); ">=</span> <span class="n" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">deferToThread</span><span class="o" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; color: rgb(102, 102, 102); ">.</span><span class="n" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">__get__</span></pre><pre style="font-size: 1em; background-color: rgb(241, 241, 241); border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; border-width: initial; border-color: initial; overflow: inherit; position: static; z-index: auto; ">....</pre><pre style="font-size: 1em; background-color: rgb(241, 241, 241); border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; border-width: initial; border-color: initial; overflow: inherit; position: static; z-index: auto; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; line-height: normal; white-space: normal; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; "><div>@deferred</div><div>def int2binAsync(anInteger):</div><div> #Packs an integer, result is 4 bytes</div><div> return struct.pack("i", anInteger) </div><div><br></div><div>@deferred</div><div>def bin2intAsync(aBin):</div><div> #Unpacks a bytestring into an integer</div><div> return struct.unpack("i", aBin)[0] </div></span></pre><pre style="font-size: 1em; background-color: rgb(241, 241, 241); border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; border-width: initial; border-color: initial; overflow: inherit; position: static; z-index: auto; "><br></pre><pre style="font-size: 1em; background-color: rgb(241, 241, 241); border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; border-width: initial; border-color: initial; overflow: inherit; position: static; z-index: auto; "><br></pre></span></font></div><div><font class="Apple-style-span" face="Helvetica, sans-serif"><br></font><div apple-content-edited="true"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div>Kind regards,</div><div><br></div><div>Valeriy Pogrebitskiy</div><div><a href="mailto:vpogrebi@verizon.net">vpogrebi@verizon.net</a></div><div><br></div></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline"> </div><br><div><div>On Oct 13, 2009, at 9:18 AM, Dirk Moors wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Hello Everyone!<div><br></div><div>My name is Dirk Moors, and since 4 years now, I've been involved in developing a cloud computing platform, using Python as the programming language. A year ago I discovered Twisted Python, and it got me very interested, upto the point where I made the decision to convert our platform (in progress) to a Twisted platform. One year later I'm still very enthousiastic about the overal performance and stability, but last week I encountered something I did't expect;</div> <div><br></div><div>It appeared that it was less efficient to run small "atomic" operations in different deferred-callbacks, when compared to running these "atomic" operations together in "blocking" mode. Am I doing something wrong here?</div> <div><br></div><div>To prove the problem to myself, I created the following example (Full source- and test code is attached):</div><div>---------------------------------------------------------------------------------------------------------------------------------------------------------------------</div> <div>import struct</div><div><br></div><div><div>def int2binAsync(anInteger):</div><div> def packStruct(i):</div><div> #Packs an integer, result is 4 bytes</div><div> return struct.pack("i", i) </div> <div><br></div><div> d = defer.Deferred()</div><div> d.addCallback(packStruct)</div><div><br></div><div> reactor.callLater(0,</div><div> d.callback,</div><div> anInteger)</div> <div><br></div><div> return d</div><div><br></div><div>def bin2intAsync(aBin): </div><div> def unpackStruct(p):</div><div> #Unpacks a bytestring into an integer</div><div> return struct.unpack("i", p)[0]</div> <div><br></div><div> d = defer.Deferred()</div><div> d.addCallback(unpackStruct)</div><div><br></div><div> reactor.callLater(0,</div><div> d.callback,</div><div> aBin)</div> <div> return d</div><div><br></div><div>def int2binSync(anInteger):</div><div> #Packs an integer, result is 4 bytes</div><div> return struct.pack("i", anInteger) </div><div><br></div><div>def bin2intSync(aBin):</div> <div> #Unpacks a bytestring into an integer</div><div> return struct.unpack("i", aBin)[0] </div><div><br></div><div>---------------------------------------------------------------------------------------------------------------------------------------------------------------------</div> <div><br></div><div>While running the testcode I got the following results:</div><div><br></div><div>(1 run = converting an integer to a byte string, converting that byte string back to an integer, and finally checking whether that last integer is the same as the input integer.)</div> <div><br></div><div><div>*** Starting Synchronous Benchmarks. <b>(No Twisted => "blocking" code)</b></div><div> -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds.</div><div> -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds.</div> <div> -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds.</div><div> -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159 seconds.</div><div> -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722 seconds.</div> <div> -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077 seconds.</div><div>*** Synchronous Benchmarks Completed in<b> 0.406000137329</b> seconds.</div><div><br></div><div><div>*** Starting Asynchronous Benchmarks . <b>(Twisted => "non-blocking" code)</b></div> <div> -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629 seconds.</div><div> -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905 seconds.</div><div> -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114 seconds.</div> <div> -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657 seconds.</div><div> -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924 seconds.</div><div> -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103 seconds.</div> <div>*** Asynchronous Benchmarks Completed in <b>42.1460001469</b> seconds.</div><div><br></div><div>Am I really seeing factor 100x??</div><div><br></div><div>I really hope that I made a huge reasoning error here but I just can't find it. If my results are correct then I really need to go and check my entire cloud platform for the places where I decided to split functions into atomic operations while thinking that it would actually improve the performance while on the contrary it did the opposit.</div> <div><br></div><div>I personaly suspect that I lose my cpu-cycles to the reactor scheduling the deferred-callbacks. Would that assumption make any sense?</div><div>The part where I need these conversion functions is in marshalling/protocol reading and writing throughout the cloud platform, which implies that these functions will be called constantly so I need them to be superfast. I always though I had to split the entire marshalling process into small atomic (deferred-callback) functions to be efficient, but these figures tell me otherwise.</div> <div><br></div><div>I really hope someone can help me out here.</div><div><br></div><div>Thanks in advance,</div><div>Best regards,</div><div>Dirk Moors</div><div><br></div><div><br></div><div><br></div><div><br></div><div> <br></div><div><br></div><div><br></div></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div><br></div><div><br></div></div> <span><twistedbenchmark.py></span>_______________________________________________<br>Twisted-Python mailing list<br><a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python<br></blockquote></div><br></div></body></html>