[Twisted-Python] Running CPU bound function concurrently in twisted

[meejah]

As a clarification to the above, parallelization of Python code across
cores is not unique to Twisted; all Python code has this same
limitation.

To use multiple cores with Python code, you need multiple Python
processes (as has been pointed out). One way to achieve this is to have
the multiple processes talking to each other (using some kind of RPC
protocol).

Another way is to simply spawn some number of subprocesses (and Twisted
has good support for running subprocesses). So, for example, if you
write a CLI tool that can be told to run "part of your problem" then
your parent Twisted process can simply spawn some number of those with
appropriate arguments to split up the problem (e.g. give each process 1
/ num_cores of the problem). This will incur some startup penalty as
each process starts up (especially if you're using PyPy, which you
should be if you care about speed) but is way simpler.

Obviously, an RPC-style communication system avoids the startup penalty
(but can be more complex).

-- 
meejah