[Twisted-Python] Running CPU bound function concurrently in twisted

[Moshe Zadka]

Hi Cheng,

deferToThread *returns* a deferred, and sends the work to a thread pool. The other two functions are useful for manipulating deferreds or other promises, but will not send work to a separate thread, so they cannot be used instead.

The simple example is pretty simple, it's pretty much

@inlineCallbacks
def some_function():
 d1 = deferToThread(cpu_heavy_square, 5)
 d2 = deferToThread(cpu_heavy_square, 6)
 five_squared, six_squared = yield gatherResults([d1, d2])
 returnValue(five_squared + six_squared)

Note that the tutorial below can be downloaded in notebook form from https://github.com/moshez/twisted-tutorial/blob/master/Twisted-Tutorial.ipynb

However, as explained, this will *defer to threads*, but will *not* use multiple cores, since Python is single-core. This is not entirely accurate: if your heavy functions are using numpy, numpy will release the GIL sometimes. Otherwise, you might want to use ampoule, but there are no easy tutorials for it that I'm aware of. You might want to consider something like dask + deferToThread for using multiple cores. See https://github.com/dask/dask-tutorial/blob/master/01_dask.delayed.ipynb for a simple example: you just plan your computation, and run the `compute` method in a thread. In this case, the only thing the thread is doing is avoiding blocking the reactor, while dask internally apportions computations.

(Thank you for the kind of words about the tutorial -- I'm one of the presenters.)

Moshe Z.

On Mon, Jun 24, 2019, at 11:34, Chengi Liu wrote:
> Thanks for pointing that out. Is there a dead simply example on how to parallelize a function using ampoule.
> Again, I have a function.. I have a for loop.. the function takes in an argument from loop.. but the for loop is extremely parallelizable.
> 
> Also, I was going thru this awesome twisted tutorial from pycon 2107 (https://www.youtube.com/watch?v=ztkBG4qLR_4)
> I found couple of use features.
> (defer.*ensureDeferred*
> and
> defer.*gatherResults*
> )
> Can these two be used instead of deferToThread.
> When are the above defered used and when to use deferToThread.
> 
> Thanks for entertaining my dumb questions.
> 
> On Mon, Jun 24, 2019 at 1:28 AM Gelin Yan <dynamicgl at gmail.com> wrote:
>> Hi
>> 
>>  If your purpose is merely not to block the reactor, you can run your code in a separate thread by using deferToThread. If you want to benefit from multi cores, you may consider use https://github.com/twisted/ampoule or other similar tools.
>> 
>> Regards
>> 
>> gelin yan
>> _______________________________________________
>>  Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20190624/4462b994/attachment-0002.html>