[Twisted-web] Parallelism is twisted

Andreas Kostyrka andreas at kostyrka.org
Wed Jan 6 22:28:07 EST 2010


Am Mittwoch, den 06.01.2010, 22:02 -0500 schrieb Glyph Lefkowitz:

> > 3) So, suppose I have one twisted reactor based process running,, I can use defferToThread as one of the way to kind of using multiple processing?
> 
> No, not really.  Threads in python are bound by the global interpreter lock and are therefore not that useful for using multiple processors.

While this is generally true, in specific cases you can use it for
speeding up by using multiple cores. One example would be a program that
does image manipulation via PIL. PIL does release the GIL before it
starts to munch on the image, hence you can achieve more than 100% user
time :)

One script that I use to scale images does manage, in ideal settings
over 300% user time; the trick being to use a couple of threads more
than I've got cpu cores, this way a couple of threads can inform the
Linux kernel what files the process is interested in. This way, the CPU
has usually enough work to do the work, and as most work is spent in C
code in PIL, the GIL does not limit CPU utilization.

But generally speaking as Glyph has mentioned, Python usually can use
only one CPU core per process. The reason for that you need to hold a
single global lock to access the virtual machine, but also the runtime
in any way.

So some C function, can do something like that:

* extract input arguments from PyObjects.
* release GIL
* munch munch
* reacquire GIL
* create PyObjects for return state.

notice that only code that in no way touches the Python runtime can run
without the GIL being hold.

Andreas

> 
> 
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web




More information about the Twisted-web mailing list