[Twisted-Python] freeing the reactor to do other jobs

[Phil Mayers]

Jeff Dyke wrote:
> I'm using the XMLRPC server in twisted and a few methods call other,
> sometimes long running, functions/methods.  I'm trying to get my brain
> around how to free the reactor to respond to other requests while this
> is happening.
> 
> A scenario.  A call is made to the server, which selects say 10K rows
> from a db and needs to check each row against a table and if they do
> not exist, insert them.
> 
> """ Oversimplified version of the process """
> def getData(self,user_id):
>     rows = self.getUserData(user_id)
>     for row in rows:
>         if self.existsInQueue(row['some_id']):
>             continue
>         else:
>              self.insertQueue(row)

You could do something like this, using twisted.internet.task.Cooperator


def batch(iterable, size):
     sourceiter = iter(iterable)
     while True:
         batchiter = itertools.islice(sourceiter, size)
         yield itertools.chain([batchiter.next()], batchiter)

coop = twisted.internet.task.Cooperator()

def _gotdata(rows):
     d = defer.Deferred()
     def worker():
         # only do 10 rows at a time, then yield control
         for rowg in batch(rows, 10):
             for row in rowg:
                 # do something
             yield
         d.callback(True)
     coop.coiterate(worker())
     return d

class foo:
     def xmlrpc_thing(self, userid):
         d = getUserData(userid)
         d.addCallback(_gotdata)
         return d

You can so similar things with defer.inlineCallbacks, or even just plain 
deferreds if you want to work hard at it.