[Twisted-Python] freeing the reactor to do other jobs

Jeff Dyke jeff.dyke at gmail.com
Fri Nov 7 08:29:50 EST 2008

I'm using the XMLRPC server in twisted and a few methods call other,
sometimes long running, functions/methods.  I'm trying to get my brain
around how to free the reactor to respond to other requests while this
is happening.

A scenario.  A call is made to the server, which selects say 10K rows
from a db and needs to check each row against a table and if they do
not exist, insert them.

""" Oversimplified version of the process """
def getData(self,user_id):
    rows = self.getUserData(user_id)
    for row in rows:
        if self.existsInQueue(row['some_id']):

I want the caller to wait on a result from this process, but I also
want the reactor to be able to handle other requests as they come in.
This function is not directly registered in the xmlrpc server with
xmlrpc_getData, but is called by that type of method after validation
that it is allowed to run in this context.

What i've seen when this has thousands of rows to process is that the
reactor is tied up and can not respond to requests until complete.
Which obviously leads to me believe that I'm not using twisted
correctly/to its potential.  I have read the deferred/asynchronous doc
pages...but am having a hard time getting my head around it and would
appreciate any advice.

When i don't specifically need the caller to get the final result,
i've been suing deferToThread, but feel in some of those instances i
could possibly write better code, rather then sending to a thread.


More information about the Twisted-Python mailing list