[Twisted-Python] twisted threading with a long running process

Jean-Paul Calderone exarkun at divmod.com
Fri Jun 27 12:37:17 EDT 2008


On Fri, 27 Jun 2008 11:46:05 -0400, Doug Farrell <dfarrell at mypublisher.com> wrote:
>Hi all,
>
>I've built a Twisted server that handles Python logging messages sent
>from the logging.handlers.SocketHandler system. Right now my protocol
>implementation handles logging the received logRecord to a file and
>entering it into an in memory model for statistical purposes. I'm
>thinking of off-loading this activity to a separate thread by having the
>protocol just send the logRecord it receives into queue that is serviced
>by a separate thread. My question is whether using the
>twisted.internet.threads.deferToThread() method is a good way to kick
>off this other thread. The other thread will be running for as long as
>the Twisted server runs so it can handle logRecords in the queue. Would
>just starting a threading.Thread based class be a better solution,
>giving access to queue inside the class to the protocol?
>

deferToThread is based on a thread-pool.  If there will be benefit to
processing log records concurrently in different threads (ie, if most
of the cost is disk I/O or an extension module that releases the GIL)
and you have multiple CPUs, using a threadpool might give you a speed
up.  If any of these things isn't true, then one thread will give you
just as much benefit as a whole pool full.  Of course, even given all
that, using deferToThread is simpler than writing a new Thread with a
queue for input, so you may still want to use deferToThread.

Jean-Paul




More information about the Twisted-Python mailing list