Hello Everyone!<div><br></div><div>My name is Dirk Moors, and since 4 years now, I've been involved in developing a cloud computing platform, using Python as the programming language. A year ago I discovered Twisted Python, and it got me very interested, upto the point where I made the decision to convert our platform (in progress) to a Twisted platform. One year later I'm still very enthousiastic about the overal performance and stability, but last week I encountered something I did't expect;</div>
<div><br></div><div>It appeared that it was less efficient to run small "atomic" operations in different deferred-callbacks, when compared to running these "atomic" operations together in "blocking" mode. Am I doing something wrong here?</div>
<div><br></div><div>To prove the problem to myself, I created the following example (Full source- and test code is attached):</div><div>---------------------------------------------------------------------------------------------------------------------------------------------------------------------</div>
<div>import struct</div><div><br></div><div><div>def int2binAsync(anInteger):</div><div> def packStruct(i):</div><div> #Packs an integer, result is 4 bytes</div><div> return struct.pack("i", i) </div>
<div><br></div><div> d = defer.Deferred()</div><div> d.addCallback(packStruct)</div><div><br></div><div> reactor.callLater(0,</div><div> d.callback,</div><div> anInteger)</div>
<div><br></div><div> return d</div><div><br></div><div>def bin2intAsync(aBin): </div><div> def unpackStruct(p):</div><div> #Unpacks a bytestring into an integer</div><div> return struct.unpack("i", p)[0]</div>
<div><br></div><div> d = defer.Deferred()</div><div> d.addCallback(unpackStruct)</div><div><br></div><div> reactor.callLater(0,</div><div> d.callback,</div><div> aBin)</div>
<div> return d</div><div><br></div><div>def int2binSync(anInteger):</div><div> #Packs an integer, result is 4 bytes</div><div> return struct.pack("i", anInteger) </div><div><br></div><div>def bin2intSync(aBin):</div>
<div> #Unpacks a bytestring into an integer</div><div> return struct.unpack("i", aBin)[0] </div><div><br></div><div>---------------------------------------------------------------------------------------------------------------------------------------------------------------------</div>
<div><br></div><div>While running the testcode I got the following results:</div><div><br></div><div>(1 run = converting an integer to a byte string, converting that byte string back to an integer, and finally checking whether that last integer is the same as the input integer.)</div>
<div><br></div><div><div>*** Starting Synchronous Benchmarks. <b>(No Twisted => "blocking" code)</b></div><div> -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds.</div><div> -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds.</div>
<div> -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds.</div><div> -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159 seconds.</div><div> -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722 seconds.</div>
<div> -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077 seconds.</div><div>*** Synchronous Benchmarks Completed in<b> 0.406000137329</b> seconds.</div><div><br></div><div><div>*** Starting Asynchronous Benchmarks . <b>(Twisted => "non-blocking" code)</b></div>
<div> -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629 seconds.</div><div> -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905 seconds.</div><div> -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114 seconds.</div>
<div> -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657 seconds.</div><div> -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924 seconds.</div><div> -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103 seconds.</div>
<div>*** Asynchronous Benchmarks Completed in <b>42.1460001469</b> seconds.</div><div><br></div><div>Am I really seeing factor 100x??</div><div><br></div><div>I really hope that I made a huge reasoning error here but I just can't find it. If my results are correct then I really need to go and check my entire cloud platform for the places where I decided to split functions into atomic operations while thinking that it would actually improve the performance while on the contrary it did the opposit.</div>
<div><br></div><div>I personaly suspect that I lose my cpu-cycles to the reactor scheduling the deferred-callbacks. Would that assumption make any sense?</div><div>The part where I need these conversion functions is in marshalling/protocol reading and writing throughout the cloud platform, which implies that these functions will be called constantly so I need them to be superfast. I always though I had to split the entire marshalling process into small atomic (deferred-callback) functions to be efficient, but these figures tell me otherwise.</div>
<div><br></div><div>I really hope someone can help me out here.</div><div><br></div><div>Thanks in advance,</div><div>Best regards,</div><div>Dirk Moors</div><div><br></div><div><br></div><div><br></div><div><br></div><div>
<br></div><div><br></div><div><br></div></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div><br></div><div><br></div></div>