[Twisted-Python] Twisted Python vs. "Blocking" Python: Weird performance on small operations.

[Reza Lotun]

Hi Dirk,

I took a look at your code sample and got the async benchmark to run
with the following values:
*** Starting Asynchronous Benchmarks.

  -> Asynchronous Benchmark (1 runs) Completed in 0.000181913375854 seconds.
  -> Asynchronous Benchmark (10 runs) Completed in 0.000736951828003 seconds.
  -> Asynchronous Benchmark (100 runs) Completed in 0.00641012191772 seconds.
  -> Asynchronous Benchmark (1000 runs) Completed in 0.0741751194 seconds.
  -> Asynchronous Benchmark (10000 runs) Completed in 0.675071001053 seconds.
  -> Asynchronous Benchmark (100000 runs) Completed in 7.29738497734 seconds.

*** Asynchronous Benchmarks Completed in 8.16032314301 seconds.

Which, though still quite a bit slower than the synchronous version,
is still much better than the 40 sec. mark that you were experiencing.
My modified version simply returned defer.succeed from your aync
block-compute functions.

i.e. Instead of your initial example:
def int2binAsync(anInteger):
    def packStruct(i):
        #Packs an integer, result is 4 bytes
        return struct.pack("i", i)

    d = defer.Deferred()
    d.addCallback(packStruct)

    reactor.callLater(0,
                      d.callback,
                      anInteger)

    return d

my version does:

def int2binAsync(anInteger):
    return defer.succeed(struct.pack('i', anInteger))

A few things to note in general however:
1) Twisted shines for block I/O operations - i.e. networking. A
compute intesive process will not necessarily yield any gains in
performance by using Twisted since the Python GIL exists (a global
lock).

2) If you are doing computations that use a C module (unforunately
struct pre 2.6 I believe doesn't use a C module), there may be a
chance that the C module releases the GIL, allowing you to do those
computations in a thread. In this case you'd be better off using
deferToThread as suggested earlier.

3) There is some (usually minimal but it exists) overhead to using
Twisted. Instead of computing a bunch of stuff serially and returning
your answer as in your sync example, you're wrapping everything up in
deferreds and starting a reactor - it's definitely going to be a bit
slower than the pure synchronous version for this case.

Hope that makes sense.

Cheers,
Reza


-- 
Reza Lotun
mobile: +44 (0)7521 310 763
email:  rlotun at gmail.com
work:   reza at tweetdeck.com
twitter: @rlotun