[Twisted-Python] Deferred Groups?
Duncan McGreggor
duncan.mcgreggor at gmail.com
Sat Jan 21 00:32:36 EST 2006
I have a question about an approach I used... I'm worried that I've
over-worked it and have over-looked a more elegant and standard
solution.
I have the need to fire off network connections in groups. Deferreds
added to a DeferredList don't fit the bill (because there's no control
over all the deferreds in the list). As an example, if you wanted to
make a whole batch of concurrent connections, but didn't want to incur
the overhead of firing off more than 20 simultaneous connections, you'd
split your destination hosts up into groups of 20. As a group was
completed, a callback could fire off the next group, etc.
What's more, I didn't want to put this kind of control in a factory or
a protocol. In my mind, that didn't seem the proper place for it...
I have some test code that generates the following output. I created a
class that instantiates a ClientFactory instance and then manages a set
of deferreds and deferred lists. Here's some sample output:
Entered main run() loop.
deferred #1 is connecting to adytum.us...
deferred #2 is connecting to del.icio.us...
deferred #3 is connecting to google.com...
Left main run() loop.
deferred #1 connected!
deferred #2 connected!
deferred #3 connected!
Finished with Group 1
deferred #4 is connecting to last.fm...
deferred #5 is connecting to washingtonpost.com...
deferred #6 is connecting to yahoo.com...
deferred #5 connected!
deferred #6 connected!
deferred #4 connected!
Finished with Group 2
deferred #7 is connecting to microsoft.com...
deferred #8 is connecting to amazon.com...
deferred #7 connected!
deferred #8 connected!
Finished with Group 3
Finished all groups.
Here are some pertinent parts of the code (full pasting:
http://pastebin.adytum.us/39/1 ):
class DeferredsByGroup(object):
[...]
def initiate(self):
self.groups = self.splitLoad(self.hosts, self.hosts_per_group)
# get the first group and let the callback handle the next one
host_group = self.groups.next()
dl = self.setupGroupDeferred(host_group)
dl.addCallback(self.getNextGroup)
dl.addErrback(self.noMoreGroups)
def splitLoad(self, host_list, per_group):
group_count, remainder = divmod(len(host_list), per_group)
if remainder: group_count += 1
for i in xrange(group_count):
yield host_list[i*per_group:i*per_group+per_group]
[...]
def getNextGroup(self, null):
group = self.groups.next()
dl = self.setupGroupDeferred(group)
dl.addCallback(self.getNextGroup)
dl.addErrback(self.noMoreGroups)
def setupHostDeferred(self, host):
self.host_counter += 1
name = "deferred #%s" % self.host_counter
f = Factory(host, name)
d = f.deferred
d.addCallback(self.updateData)
return d
def setupGroupDeferred(self, group):
self.group_counter += 1
# iterate through each host in the group
deferreds = [ self.setupHostDeferred(host) for host in group ]
# set up this group as a deferred list
dl = defer.DeferredList(deferreds)
dl.addCallback(self.handleGroup)
dl.addErrback(self.handleFailure)
return dl
[...]
And this is instantiated in the following manner:
hosts = [
'adytum.us',
'del.icio.us',
'google.com',
'last.fm',
'washingtonpost.com',
'yahoo.com',
'microsoft.com',
'amazon.com',
]
hosts_per_group = 3
runner = DeferredsByGroup(hosts, hosts_per_group)
runner.initiate()
reactor.run()
More information about the Twisted-Python
mailing list