[Twisted-Python] freeing the reactor to do other jobs
Phil Mayers
p.mayers at imperial.ac.uk
Fri Nov 7 09:00:14 EST 2008
Jeff Dyke wrote:
> I'm using the XMLRPC server in twisted and a few methods call other,
> sometimes long running, functions/methods. I'm trying to get my brain
> around how to free the reactor to respond to other requests while this
> is happening.
>
> A scenario. A call is made to the server, which selects say 10K rows
> from a db and needs to check each row against a table and if they do
> not exist, insert them.
>
> """ Oversimplified version of the process """
> def getData(self,user_id):
> rows = self.getUserData(user_id)
> for row in rows:
> if self.existsInQueue(row['some_id']):
> continue
> else:
> self.insertQueue(row)
You could do something like this, using twisted.internet.task.Cooperator
def batch(iterable, size):
sourceiter = iter(iterable)
while True:
batchiter = itertools.islice(sourceiter, size)
yield itertools.chain([batchiter.next()], batchiter)
coop = twisted.internet.task.Cooperator()
def _gotdata(rows):
d = defer.Deferred()
def worker():
# only do 10 rows at a time, then yield control
for rowg in batch(rows, 10):
for row in rowg:
# do something
yield
d.callback(True)
coop.coiterate(worker())
return d
class foo:
def xmlrpc_thing(self, userid):
d = getUserData(userid)
d.addCallback(_gotdata)
return d
You can so similar things with defer.inlineCallbacks, or even just plain
deferreds if you want to work hard at it.
More information about the Twisted-Python
mailing list