[Twisted-Python] Re: CPU intensive threads
natester at gmail.com
Wed Jul 27 19:02:39 EDT 2005
These CPU intensive threads are python/C extensions generated with
pyrex. Instead of messing with the GIL I will use processes as David
suggests. It would be nice to have a process pool implementation that
was as easy to use as deferToThread.
Evidently they use the perspective broker for inter process
communication in quotient.searchup, and that may be a good starting
On 27 Jul 2005 17:10:42 -0400, David Bolen <db3l at fitlinxx.com> wrote:
> Jp Calderone <exarkun at divmod.com> writes:
> > POSH doesn't elimate the transport overhead. I've done some basic
> > investigation, and it's incredibly slow. mmap() is probably a
> > better solution in most cases, although I am not convinced multiple
> > processes are called for in this case.
> Me neither, but it would be a logical way to progress if you couldn't
> resolve things in the single process. Good to know about POSH though.
> > A solution which hasn't been suggested yet is to drop the native
> > thread and use a cooperative Python thread. With this approach, you
> > can choose to schedule it however you like, including /not/
> > scheduling it when you have other more important tasks to complete.
> Well, but you'd still have the problem of ensuring that it was
> yielding back at a reasonable frequency wouldn't you? So it would be
> similar to sprinkling in a time.sleep(0) in a non-cooperative thread,
> and still subject to cases where it might not be that simple.
> -- David
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
More information about the Twisted-Python