[Twisted-Python] hosting cpu intensive algorithms with twisted. Is it worth it?
James Y Knight
foom at fuhm.net
Thu Aug 5 13:40:23 EDT 2004
On Aug 4, 2004, at 11:12 AM, Kelly Dyer wrote:
> I have a number of cpu-heavy signal processing routines that I want to
> make remotely available.
> Is it worth the energy to use the twisted framework in order to build
> an application like this? I have a cluster available and was hoping
> to use its thread-migration to even out the processing and provide
> If you feel this is a do-able application, any design suggestions for
> an excited twisted neophyte would be welcome!
Are you going to be writing your routines in python? If so, you cannot
use threads, as python only allows one thread at a time to run python
bytecode, thus making any CPUs above one useless. Also, what clustering
software are you using? The ones I've seen used in the past only allow
migration of unthreaded programs, as shared memory consistency across a
network is hard (and/or slow).
What you can do, though, (and something I would generally recommend,
although I don't know the specifics of your problem), is to spawn off
multiple *processes* to handle the long running routines, and use
Twisted to coordinate the network and process IO.
If you are not writing your routines in python and your cluster
software actually does support migration of threads, you can probably
get away with running threads in twisted.
More information about the Twisted-Python