[Twisted-Python] Re: CPU intensive threads
exarkun at divmod.com
Wed Jul 27 16:19:43 EDT 2005
On 27 Jul 2005 11:05:43 -0400, David Bolen <db3l at fitlinxx.com> wrote:
>Nathaniel Haggard <natester at gmail.com> writes:
>> Is there a way to set the priority of the main part of twisted so that
>> it can run CPU intensive threads and still service connections.
>If you can't afford the time to transfer requests and/or results to a
>separate process over a normal channel, and are on a Posix platform,
>you might also investigate POSH (http://poshmodule.sourceforge.net),
>which implements object sharing in shared memory between processes,
>which would eliminate the transport overhead but still let you
>separate the processing into distinct processes. It's an early
>development project that I've only experimented with but if it fits
>your needs it might do well. (It comes with a simple
>producer/consumer example that could probably be used as a starting
POSH doesn't elimate the transport overhead. I've done some basic investigation, and it's incredibly slow. mmap() is probably a better solution in most cases, although I am not convinced multiple processes are called for in this case.
A solution which hasn't been suggested yet is to drop the native thread and use a cooperative Python thread. With this approach, you can choose to schedule it however you like, including /not/ scheduling it when you have other more important tasks to complete.
More information about the Twisted-Python