Hello Reza,<div><br></div><div>I tried the solution you provided and I have to say, that changed a lot!</div><div>You gave me a better understanding of how things work with Twisted, and I really appreciate your response!</div>
<div><br></div><div>Thanks for your help!</div><div>Best regards,</div><div>Dirk Moors </div><div><br><div class="gmail_quote">2009/10/13 <span dir="ltr"><<a href="mailto:twisted-python-request@twistedmatrix.com">twisted-python-request@twistedmatrix.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Send Twisted-Python mailing list submissions to<br>
<a href="mailto:twisted-python@twistedmatrix.com">twisted-python@twistedmatrix.com</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:twisted-python-request@twistedmatrix.com">twisted-python-request@twistedmatrix.com</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:twisted-python-owner@twistedmatrix.com">twisted-python-owner@twistedmatrix.com</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Twisted-Python digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Twisted Python vs. "Blocking" Python: Weird performance<br>
on small operations. (Reza Lotun)<br>
2. Re: Twisted-Python Digest, Vol 67, Issue 22 (Dirk Moors)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 13 Oct 2009 15:04:06 +0100<br>
From: Reza Lotun <<a href="mailto:rlotun@gmail.com">rlotun@gmail.com</a>><br>
Subject: Re: [Twisted-Python] Twisted Python vs. "Blocking" Python:<br>
Weird performance on small operations.<br>
To: Twisted general discussion <<a href="mailto:twisted-python@twistedmatrix.com">twisted-python@twistedmatrix.com</a>><br>
Message-ID:<br>
<<a href="mailto:95bb10690910130704o7c0ff2besf00dcf5918990dcf@mail.gmail.com">95bb10690910130704o7c0ff2besf00dcf5918990dcf@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=UTF-8<br>
<br>
Hi Dirk,<br>
<br>
I took a look at your code sample and got the async benchmark to run<br>
with the following values:<br>
*** Starting Asynchronous Benchmarks.<br>
<br>
-> Asynchronous Benchmark (1 runs) Completed in 0.000181913375854 seconds.<br>
-> Asynchronous Benchmark (10 runs) Completed in 0.000736951828003 seconds.<br>
-> Asynchronous Benchmark (100 runs) Completed in 0.00641012191772 seconds.<br>
-> Asynchronous Benchmark (1000 runs) Completed in 0.0741751194 seconds.<br>
-> Asynchronous Benchmark (10000 runs) Completed in 0.675071001053 seconds.<br>
-> Asynchronous Benchmark (100000 runs) Completed in 7.29738497734 seconds.<br>
<br>
*** Asynchronous Benchmarks Completed in 8.16032314301 seconds.<br>
<br>
Which, though still quite a bit slower than the synchronous version,<br>
is still much better than the 40 sec. mark that you were experiencing.<br>
My modified version simply returned defer.succeed from your aync<br>
block-compute functions.<br>
<br>
i.e. Instead of your initial example:<br>
def int2binAsync(anInteger):<br>
def packStruct(i):<br>
#Packs an integer, result is 4 bytes<br>
return struct.pack("i", i)<br>
<br>
d = defer.Deferred()<br>
d.addCallback(packStruct)<br>
<br>
reactor.callLater(0,<br>
d.callback,<br>
anInteger)<br>
<br>
return d<br>
<br>
my version does:<br>
<br>
def int2binAsync(anInteger):<br>
return defer.succeed(struct.pack('i', anInteger))<br>
<br>
A few things to note in general however:<br>
1) Twisted shines for block I/O operations - i.e. networking. A<br>
compute intesive process will not necessarily yield any gains in<br>
performance by using Twisted since the Python GIL exists (a global<br>
lock).<br>
<br>
2) If you are doing computations that use a C module (unforunately<br>
struct pre 2.6 I believe doesn't use a C module), there may be a<br>
chance that the C module releases the GIL, allowing you to do those<br>
computations in a thread. In this case you'd be better off using<br>
deferToThread as suggested earlier.<br>
<br>
3) There is some (usually minimal but it exists) overhead to using<br>
Twisted. Instead of computing a bunch of stuff serially and returning<br>
your answer as in your sync example, you're wrapping everything up in<br>
deferreds and starting a reactor - it's definitely going to be a bit<br>
slower than the pure synchronous version for this case.<br>
<br>
Hope that makes sense.<br>
<br>
Cheers,<br>
Reza<br>
<br>
<br>
--<br>
Reza Lotun<br>
mobile: +44 (0)7521 310 763<br>
email: <a href="mailto:rlotun@gmail.com">rlotun@gmail.com</a><br>
work: <a href="mailto:reza@tweetdeck.com">reza@tweetdeck.com</a><br>
twitter: @rlotun<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Tue, 13 Oct 2009 16:18:35 +0200<br>
From: Dirk Moors <<a href="mailto:dirkmoors@gmail.com">dirkmoors@gmail.com</a>><br>
Subject: Re: [Twisted-Python] Twisted-Python Digest, Vol 67, Issue 22<br>
To: <a href="mailto:twisted-python@twistedmatrix.com">twisted-python@twistedmatrix.com</a><br>
Message-ID:<br>
<<a href="mailto:cf75a1410910130718m53645515oc65f0890366a12f2@mail.gmail.com">cf75a1410910130718m53645515oc65f0890366a12f2@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hello Valeriy,<br>
<br>
I tried the thing you suggested, and I attached the (updated) code.<br>
Unfortunatly, the new code was even slower, producing the following results:<br>
<br>
*** Starting Asynchronous Benchmarks. (Using Twisted, with<br>
"deferred-decorator")<br>
-> Asynchronous Benchmark (1 runs) Completed in 56.0279998779 seconds.<br>
-> Asynchronous Benchmark (10 runs) Completed in 56.0130000114 seconds.<br>
-> Asynchronous Benchmark (100 runs) Completed in 56.010999918 seconds.<br>
-> Asynchronous Benchmark (1000 runs) Completed in 56.0410001278 seconds.<br>
-> Asynchronous Benchmark (10000 runs) Completed in 56.3069999218 seconds.<br>
-> Asynchronous Benchmark (100000 runs) Completed in 58.8910000324<br>
seconds.<br>
*** Asynchronous Benchmarks Completed in 59.4659998417 seconds.<br>
<br>
I suspect that this would me more inefficient because with the deferToThread<br>
function in place, every single operation will be executed in its own<br>
thread, which means:<br>
(1 x 2) + (10 x 2) + (100 x 2) + (1000 x 2) + (10000 x 2) + (100000 x 2)<br>
threads....which is...a lot.<br>
<br>
Maybe the problem lies in the way I test the code? I understand that using<br>
the asynchronous testcode this way (generating the deferreds using a<br>
FOR-loop), a lot of deferreds are generated before the reactor starts<br>
calling the deferred-callbacks.....would there be another, better way to<br>
test the code?<br>
The reason I need to now which one is faster (async vs sync functions) is<br>
because I need to decide on whetehr or not I should re-evaluate the code I<br>
just recently finished building.<br>
<br>
Any other ideas maybe?<br>
<br>
Thanks in advance,<br>
Dirk<br>
<br>
<br>
________________________________________________________________________________________________________________________________________________________<br>
> Message: 3<br>
> Date: Tue, 13 Oct 2009 09:41:19 -0400<br>
> From: Valeriy Pogrebitskiy <<a href="mailto:vpogrebi@verizon.net">vpogrebi@verizon.net</a>><br>
> Subject: Re: [Twisted-Python] Twisted Python vs. "Blocking" Python:<br>
> Weird performance on small operations.<br>
> To: Twisted general discussion <<a href="mailto:twisted-python@twistedmatrix.com">twisted-python@twistedmatrix.com</a>><br>
> Message-ID: <<a href="mailto:EDB2B354-B25D-4A98-AC9D-B9745CA6C3AB@verizon.net">EDB2B354-B25D-4A98-AC9D-B9745CA6C3AB@verizon.net</a>><br>
> Content-Type: text/plain; charset="us-ascii"<br>
><br>
> Dirk,<br>
><br>
> Using deferred directly in your bin2intAsync() may be somewhat less<br>
> efficient than some other way described in Recipe 439358: [Twisted]<br>
> From blocking functions to deferred functions<br>
><br>
> recipe (<a href="http://code.activestate.com/recipes/439358/" target="_blank">http://code.activestate.com/recipes/439358/</a>)<br>
><br>
> You would get same effect (asynchronous execution) - but potentially<br>
> more efficiently - by just decorating your synchronous methods as:<br>
><br>
> from twisted.internet.threads import deferToThread<br>
> deferred = deferToThread.__get__<br>
> ....<br>
> @deferred<br>
> def int2binAsync(anInteger):<br>
> #Packs an integer, result is 4 bytes<br>
> return struct.pack("i", anInteger)<br>
><br>
> @deferred<br>
> def bin2intAsync(aBin):<br>
> #Unpacks a bytestring into an integer<br>
> return struct.unpack("i", aBin)[0]<br>
><br>
><br>
><br>
><br>
> Kind regards,<br>
><br>
> Valeriy Pogrebitskiy<br>
> <a href="mailto:vpogrebi@verizon.net">vpogrebi@verizon.net</a><br>
><br>
><br>
><br>
><br>
> On Oct 13, 2009, at 9:18 AM, Dirk Moors wrote:<br>
><br>
> > Hello Everyone!<br>
> ><br>
> > My name is Dirk Moors, and since 4 years now, I've been involved in<br>
> > developing a cloud computing platform, using Python as the<br>
> > programming language. A year ago I discovered Twisted Python, and it<br>
> > got me very interested, upto the point where I made the decision to<br>
> > convert our platform (in progress) to a Twisted platform. One year<br>
> > later I'm still very enthousiastic about the overal performance and<br>
> > stability, but last week I encountered something I did't expect;<br>
> ><br>
> > It appeared that it was less efficient to run small "atomic"<br>
> > operations in different deferred-callbacks, when compared to running<br>
> > these "atomic" operations together in "blocking" mode. Am I doing<br>
> > something wrong here?<br>
> ><br>
> > To prove the problem to myself, I created the following example<br>
> > (Full source- and test code is attached):<br>
> ><br>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> > import struct<br>
> ><br>
> > def int2binAsync(anInteger):<br>
> > def packStruct(i):<br>
> > #Packs an integer, result is 4 bytes<br>
> > return struct.pack("i", i)<br>
> ><br>
> > d = defer.Deferred()<br>
> > d.addCallback(packStruct)<br>
> ><br>
> > reactor.callLater(0,<br>
> > d.callback,<br>
> > anInteger)<br>
> ><br>
> > return d<br>
> ><br>
> > def bin2intAsync(aBin):<br>
> > def unpackStruct(p):<br>
> > #Unpacks a bytestring into an integer<br>
> > return struct.unpack("i", p)[0]<br>
> ><br>
> > d = defer.Deferred()<br>
> > d.addCallback(unpackStruct)<br>
> ><br>
> > reactor.callLater(0,<br>
> > d.callback,<br>
> > aBin)<br>
> > return d<br>
> ><br>
> > def int2binSync(anInteger):<br>
> > #Packs an integer, result is 4 bytes<br>
> > return struct.pack("i", anInteger)<br>
> ><br>
> > def bin2intSync(aBin):<br>
> > #Unpacks a bytestring into an integer<br>
> > return struct.unpack("i", aBin)[0]<br>
> ><br>
> ><br>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> ><br>
> > While running the testcode I got the following results:<br>
> ><br>
> > (1 run = converting an integer to a byte string, converting that<br>
> > byte string back to an integer, and finally checking whether that<br>
> > last integer is the same as the input integer.)<br>
> ><br>
> > *** Starting Synchronous Benchmarks. (No Twisted => "blocking" code)<br>
> > -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds.<br>
> > -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds.<br>
> > -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds.<br>
> > -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159<br>
> > seconds.<br>
> > -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722<br>
> > seconds.<br>
> > -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077<br>
> > seconds.<br>
> > *** Synchronous Benchmarks Completed in 0.406000137329 seconds.<br>
> ><br>
> > *** Starting Asynchronous Benchmarks . (Twisted => "non-blocking"<br>
> > code)<br>
> > -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629<br>
> > seconds.<br>
> > -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905<br>
> > seconds.<br>
> > -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114<br>
> > seconds.<br>
> > -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657<br>
> > seconds.<br>
> > -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924<br>
> > seconds.<br>
> > -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103<br>
> > seconds.<br>
> > *** Asynchronous Benchmarks Completed in 42.1460001469 seconds.<br>
> ><br>
> > Am I really seeing factor 100x??<br>
> ><br>
> > I really hope that I made a huge reasoning error here but I just<br>
> > can't find it. If my results are correct then I really need to go<br>
> > and check my entire cloud platform for the places where I decided to<br>
> > split functions into atomic operations while thinking that it would<br>
> > actually improve the performance while on the contrary it did the<br>
> > opposit.<br>
> ><br>
> > I personaly suspect that I lose my cpu-cycles to the reactor<br>
> > scheduling the deferred-callbacks. Would that assumption make any<br>
> > sense?<br>
> > The part where I need these conversion functions is in marshalling/<br>
> > protocol reading and writing throughout the cloud platform, which<br>
> > implies that these functions will be called constantly so I need<br>
> > them to be superfast. I always though I had to split the entire<br>
> > marshalling process into small atomic (deferred-callback) functions<br>
> > to be efficient, but these figures tell me otherwise.<br>
> ><br>
> > I really hope someone can help me out here.<br>
> ><br>
> > Thanks in advance,<br>
> > Best regards,<br>
> > Dirk Moors<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > <twistedbenchmark.py>_______________________________________________<br>
> > Twisted-Python mailing list<br>
> > <a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
> > <a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
><br>
> -------------- next part --------------<br>
> An HTML attachment was scrubbed...<br>
> URL:<br>
> <a href="http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/e9ae2546/attachment.htm" target="_blank">http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/e9ae2546/attachment.htm</a><br>
><br>
> ------------------------------<br>
><br>
> _______________________________________________<br>
> Twisted-Python mailing list<br>
> <a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
> <a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
><br>
><br>
> End of Twisted-Python Digest, Vol 67, Issue 22<br>
> **********************************************<br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/357ffe0c/attachment.htm" target="_blank">http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/357ffe0c/attachment.htm</a><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: twistedbenchmark.py<br>
Type: application/octet-stream<br>
Size: 7269 bytes<br>
Desc: not available<br>
Url : <a href="http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/357ffe0c/attachment.obj" target="_blank">http://twistedmatrix.com/pipermail/twisted-python/attachments/20091013/357ffe0c/attachment.obj</a><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
<br>
<br>
End of Twisted-Python Digest, Vol 67, Issue 23<br>
**********************************************<br>
</blockquote></div><br></div>