[Twisted-Python] Interleaving long running loops

Umit Oztosun umit at oztosun.net
Wed Nov 2 06:40:56 EDT 2011


Hello,

I have been developing a medium/large scale application using twisted
and very happy about it. It is composed of two parts, an application
server and a PyQt based client. Perspective broker is used for all
communication between client and server. Server mainly talks with a
database, formats results and sends to clients (roughly).

Most formatting is done in simple for loops (there are many different
kinds of operations which a client can choose):

@defer.inlineCallbacks
def operation():
    dbQueryResult = yield runSomeSQL()
    for item in dbQueryResult:
        process(item)

Again, most of the time these are light operations which complete in
sub-seconds. However, rarely, results may be so large that it takes
several seconds to process, and during this period all other server
processing stops (i.e. accepting new client connections).

I've written a utility function, deferredSleep(), which I call
occasionally to avoid blocking:

def deferredSleep(seconds=0.01):
    d = defer.Deferred()
    reactor.callLater(seconds, d.callback, None)
    return d

And I use it like below to avoid blocking for very large result sets:

@defer.inlineCallbacks
def operation():
    dbQueryResult = yield runSomeSQL()
    for i, item in enumerate(dbQueryResult):
        if i % 1000 == 0:
            # Breathe after a thousand iterations
            yield deferredSleep(0.01)
        process(item)

So my question is, is this an acceptable solution? And is there a more
generic way, i.e.

for item in magicalIterator(dbQueryResult):
    process(item)

I have lots of similar loops all over the place and a more generic
solution would be great. I am also aware of deferToThread but I am
trying to avoid using threads.

Regards,
--
Umit



More information about the Twisted-Python mailing list