[Twisted-Python] Running CPU bound function concurrently in twisted

[Moshe Zadka]

On Mon, Jun 24, 2019, at 17:24, Chengi Liu wrote:
> How do I use more multiple CPU cores? I guess atleast the fact that twisted app wont be blocking is good, but what would be a good way to use multiple cores.

It *really* depends on what is the function that is CPU heavy, and why it is CPU heavy.
You have a lot of options here, and they're all subtly different.

But I'll start with noting that this is *no longer* a Twisted question: using multiple cores from Python is genuinely a problem for which there is no single good solution, just a number of different ones with trade-offs. As I mentioned, you can use `dask`. I think that might be the simplest solution for you -- dask has a lot of good tutorials, and if your only goal is using multiple CPUs, it has an "auto-setup". Another option is to see if you can use a library that natively releases the Global Interpreter Lock. If, for example, your function is numeric computation heavy, than numpy does release the GIL. You can also use ampoule, though the set-up becomes non-trivial -- you will need to either change your process initialization or maintain a process pool from the main process. Either option can become non-trivial fast. Finally, the option of writing the CPU heavy code in Cython, and then releasing the GIL explicitly around the area it is not needed, exists.

To sum up:

* Dask
* Something that releases the GIL, like numpy
* Setting up an ampoule process pool
* Move CPU heavy logic to Cython, and release GIL explicitly

As I said, none of these options are really Twisted-related (the only one vaguely connected is the ampoule one, but even then, the fact that you'll use Twisted to manage the pool is incidental). There are even more exotic options, of course, but I think this e-mail might already be confusing enough.

Moshe Z.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20190624/67bc2496/attachment-0002.html>