<div class="gmail_quote"><div>Hello Valeriy,</div><div><br></div><div>I tried the thing you suggested, and I attached the (updated) code.</div><div>Unfortunatly, the new code was even slower, producing the following results:</div>
<div><br></div><div><div>*** Starting Asynchronous Benchmarks. (Using Twisted, with "deferred-decorator")</div><div> -> Asynchronous Benchmark (1 runs) Completed in 56.0279998779 seconds.</div><div> -> Asynchronous Benchmark (10 runs) Completed in 56.0130000114 seconds.</div>
<div> -> Asynchronous Benchmark (100 runs) Completed in 56.010999918 seconds.</div><div> -> Asynchronous Benchmark (1000 runs) Completed in 56.0410001278 seconds.</div><div> -> Asynchronous Benchmark (10000 runs) Completed in 56.3069999218 seconds.</div>
<div> -> Asynchronous Benchmark (100000 runs) Completed in 58.8910000324 seconds.</div><div>*** Asynchronous Benchmarks Completed in 59.4659998417 seconds.</div><div><br></div><div>I suspect that this would me more inefficient because with the deferToThread function in place, every single operation will be executed in its own thread, which means:</div>
<div>(1 x 2) + (10 x 2) + (100 x 2) + (1000 x 2) + (10000 x 2) + (100000 x 2) threads....which is...a lot.</div><div><br></div><div>Maybe the problem lies in the way I test the code? I understand that using the asynchronous testcode this way (generating the deferreds using a FOR-loop), a lot of deferreds are generated before the reactor starts calling the deferred-callbacks.....would there be another, better way to test the code? </div>
<div>The reason I need to now which one is faster (async vs sync functions) is because I need to decide on whetehr or not I should re-evaluate the code I just recently finished building.</div><div><br></div><div>Any other ideas maybe?</div>
<div><br></div><div>Thanks in advance,</div><div>Dirk </div><div><br></div><div><br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">________________________________________________________________________________________________________________________________________________________<br>
Message: 3<br>
Date: Tue, 13 Oct 2009 09:41:19 -0400<br>
From: Valeriy Pogrebitskiy <<a href="mailto:vpogrebi@verizon.net">vpogrebi@verizon.net</a>><br>
Subject: Re: [Twisted-Python] Twisted Python vs. "Blocking" Python:<br>
Weird performance on small operations.<br>
To: Twisted general discussion <<a href="mailto:twisted-python@twistedmatrix.com">twisted-python@twistedmatrix.com</a>><br>
Message-ID: <<a href="mailto:EDB2B354-B25D-4A98-AC9D-B9745CA6C3AB@verizon.net">EDB2B354-B25D-4A98-AC9D-B9745CA6C3AB@verizon.net</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Dirk,<br>
<br>
Using deferred directly in your bin2intAsync() may be somewhat less<br>
efficient than some other way described in Recipe 439358: [Twisted]<br>
From blocking functions to deferred functions<br>
<br>
recipe (<a href="http://code.activestate.com/recipes/439358/" target="_blank">http://code.activestate.com/recipes/439358/</a>)<br>
<br>
You would get same effect (asynchronous execution) - but potentially<br>
more efficiently - by just decorating your synchronous methods as:<br>
<br>
from twisted.internet.threads import deferToThread<br>
deferred = deferToThread.__get__<br>
....<br>
@deferred<br>
def int2binAsync(anInteger):<br>
#Packs an integer, result is 4 bytes<br>
return struct.pack("i", anInteger)<br>
<br>
@deferred<br>
def bin2intAsync(aBin):<br>
#Unpacks a bytestring into an integer<br>
return struct.unpack("i", aBin)[0]<br>
<br>
<br>
<br>
<br>
Kind regards,<br>
<br>
Valeriy Pogrebitskiy<br>
<a href="mailto:vpogrebi@verizon.net">vpogrebi@verizon.net</a><br>
<br>
<br>
<br>
<br>
On Oct 13, 2009, at 9:18 AM, Dirk Moors wrote:<br>
<br>
> Hello Everyone!<br>
><br>
> My name is Dirk Moors, and since 4 years now, I've been involved in<br>
> developing a cloud computing platform, using Python as the<br>
> programming language. A year ago I discovered Twisted Python, and it<br>
> got me very interested, upto the point where I made the decision to<br>
> convert our platform (in progress) to a Twisted platform. One year<br>
> later I'm still very enthousiastic about the overal performance and<br>
> stability, but last week I encountered something I did't expect;<br>
><br>
> It appeared that it was less efficient to run small "atomic"<br>
> operations in different deferred-callbacks, when compared to running<br>
> these "atomic" operations together in "blocking" mode. Am I doing<br>
> something wrong here?<br>
><br>
> To prove the problem to myself, I created the following example<br>
> (Full source- and test code is attached):<br>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> import struct<br>
><br>
> def int2binAsync(anInteger):<br>
> def packStruct(i):<br>
> #Packs an integer, result is 4 bytes<br>
> return struct.pack("i", i)<br>
><br>
> d = defer.Deferred()<br>
> d.addCallback(packStruct)<br>
><br>
> reactor.callLater(0,<br>
> d.callback,<br>
> anInteger)<br>
><br>
> return d<br>
><br>
> def bin2intAsync(aBin):<br>
> def unpackStruct(p):<br>
> #Unpacks a bytestring into an integer<br>
> return struct.unpack("i", p)[0]<br>
><br>
> d = defer.Deferred()<br>
> d.addCallback(unpackStruct)<br>
><br>
> reactor.callLater(0,<br>
> d.callback,<br>
> aBin)<br>
> return d<br>
><br>
> def int2binSync(anInteger):<br>
> #Packs an integer, result is 4 bytes<br>
> return struct.pack("i", anInteger)<br>
><br>
> def bin2intSync(aBin):<br>
> #Unpacks a bytestring into an integer<br>
> return struct.unpack("i", aBin)[0]<br>
><br>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
><br>
> While running the testcode I got the following results:<br>
><br>
> (1 run = converting an integer to a byte string, converting that<br>
> byte string back to an integer, and finally checking whether that<br>
> last integer is the same as the input integer.)<br>
><br>
> *** Starting Synchronous Benchmarks. (No Twisted => "blocking" code)<br>
> -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds.<br>
> -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds.<br>
> -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds.<br>
> -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159<br>
> seconds.<br>
> -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722<br>
> seconds.<br>
> -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077<br>
> seconds.<br>
> *** Synchronous Benchmarks Completed in 0.406000137329 seconds.<br>
><br>
> *** Starting Asynchronous Benchmarks . (Twisted => "non-blocking"<br>
> code)<br>
> -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629<br>
> seconds.<br>
> -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905<br>
> seconds.<br>
> -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114<br>
> seconds.<br>
> -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657<br>
> seconds.<br>
> -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924<br>
> seconds.<br>
> -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103<br>
> seconds.<br>
> *** Asynchronous Benchmarks Completed in 42.1460001469 seconds.<br>
><br>
> Am I really seeing factor 100x??<br>
><br>
> I really hope that I made a huge reasoning error here but I just<br>
> can't find it. If my results are correct then I really need to go<br>
> and check my entire cloud platform for the places where I decided to<br>
> split functions into atomic operations while thinking that it would<br>
> actually improve the performance while on the contrary it did the<br>
> opposit.<br>
><br>
> I personaly suspect that I lose my cpu-cycles to the reactor<br>
> scheduling the deferred-callbacks. Would that assumption make any<br>
> sense?<br>
> The part where I need these conversion functions is in marshalling/<br>
> protocol reading and writing throughout the cloud platform, which<br>
> implies that these functions will be called constantly so I need<br>
> them to be superfast. I always though I had to split the entire<br>
> marshalling process into small atomic (deferred-callback) functions<br>
> to be efficient, but these figures tell me otherwise.<br>
><br>
> I really hope someone can help me out here.<br>
><br>
> Thanks in advance,<br>
> Best regards,<br>
> Dirk Moors<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> <twistedbenchmark.py>_______________________________________________<br>
> Twisted-Python mailing list<br>
> <a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
> <a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/e9ae2546/attachment.htm" target="_blank">http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/e9ae2546/attachment.htm</a><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
<br>
<br>
End of Twisted-Python Digest, Vol 67, Issue 22<br>
**********************************************<br>
</blockquote></div><br>