[Twisted-Python] freeing the reactor to do other jobs

Phil Mayers p.mayers at imperial.ac.uk
Fri Nov 7 09:00:14 EST 2008

Jeff Dyke wrote:
> I'm using the XMLRPC server in twisted and a few methods call other,
> sometimes long running, functions/methods.  I'm trying to get my brain
> around how to free the reactor to respond to other requests while this
> is happening.
> A scenario.  A call is made to the server, which selects say 10K rows
> from a db and needs to check each row against a table and if they do
> not exist, insert them.
> """ Oversimplified version of the process """
> def getData(self,user_id):
>     rows = self.getUserData(user_id)
>     for row in rows:
>         if self.existsInQueue(row['some_id']):
>             continue
>         else:
>              self.insertQueue(row)

You could do something like this, using twisted.internet.task.Cooperator

def batch(iterable, size):
     sourceiter = iter(iterable)
     while True:
         batchiter = itertools.islice(sourceiter, size)
         yield itertools.chain([batchiter.next()], batchiter)

coop = twisted.internet.task.Cooperator()

def _gotdata(rows):
     d = defer.Deferred()
     def worker():
         # only do 10 rows at a time, then yield control
         for rowg in batch(rows, 10):
             for row in rowg:
                 # do something
     return d

class foo:
     def xmlrpc_thing(self, userid):
         d = getUserData(userid)
         return d

You can so similar things with defer.inlineCallbacks, or even just plain 
deferreds if you want to work hard at it.

More information about the Twisted-Python mailing list