[Twisted-Python] Deferred Groups?

Duncan McGreggor duncan.mcgreggor at gmail.com
Sat Jan 21 00:32:36 EST 2006


I have a question about an approach I used... I'm worried that I've 
over-worked it and have over-looked a more elegant and standard 
solution.

I have the need to fire off network connections in groups. Deferreds 
added to a DeferredList don't fit the bill (because there's no control 
over all the deferreds in the list). As an example, if you wanted to 
make a whole batch of concurrent connections, but didn't want to incur 
the overhead of firing off more than 20 simultaneous connections, you'd 
split your destination hosts up into groups of 20. As a group was 
completed, a callback could fire off the next group, etc.

What's more, I didn't want to put this kind of control in a factory or 
a protocol. In my mind, that didn't seem the proper place for it...

I have some test code that generates the following output. I created a 
class that instantiates a ClientFactory instance and then manages a set 
of deferreds and deferred lists. Here's some sample output:

Entered main run() loop.
deferred #1 is connecting to adytum.us...
deferred #2 is connecting to del.icio.us...
deferred #3 is connecting to google.com...
Left main run() loop.
deferred #1 connected!
deferred #2 connected!
deferred #3 connected!
Finished with Group 1
deferred #4 is connecting to last.fm...
deferred #5 is connecting to washingtonpost.com...
deferred #6 is connecting to yahoo.com...
deferred #5 connected!
deferred #6 connected!
deferred #4 connected!
Finished with Group 2
deferred #7 is connecting to microsoft.com...
deferred #8 is connecting to amazon.com...
deferred #7 connected!
deferred #8 connected!
Finished with Group 3
Finished all groups.

Here are some pertinent parts of the code (full pasting: 
http://pastebin.adytum.us/39/1 ):

class DeferredsByGroup(object):
[...]
     def initiate(self):
         self.groups = self.splitLoad(self.hosts, self.hosts_per_group)
         # get the first group and let the callback handle the next one
         host_group = self.groups.next()
         dl = self.setupGroupDeferred(host_group)
         dl.addCallback(self.getNextGroup)
         dl.addErrback(self.noMoreGroups)

     def splitLoad(self, host_list, per_group):
         group_count, remainder = divmod(len(host_list), per_group)
         if remainder: group_count += 1
         for i in xrange(group_count):
             yield host_list[i*per_group:i*per_group+per_group]
[...]
     def getNextGroup(self, null):
         group = self.groups.next()
         dl = self.setupGroupDeferred(group)
         dl.addCallback(self.getNextGroup)
         dl.addErrback(self.noMoreGroups)

     def setupHostDeferred(self, host):
         self.host_counter += 1
         name = "deferred #%s" % self.host_counter
         f = Factory(host, name)
         d = f.deferred
         d.addCallback(self.updateData)
         return d

     def setupGroupDeferred(self, group):
         self.group_counter += 1
         # iterate through each host in the group
         deferreds = [ self.setupHostDeferred(host) for host in group ]
         # set up this group as a deferred list
         dl = defer.DeferredList(deferreds)
         dl.addCallback(self.handleGroup)
         dl.addErrback(self.handleFailure)
         return dl
[...]

And this is instantiated in the following manner:

hosts = [
         'adytum.us',
         'del.icio.us',
         'google.com',
         'last.fm',
         'washingtonpost.com',
         'yahoo.com',
         'microsoft.com',
         'amazon.com',
]
hosts_per_group = 3
runner = DeferredsByGroup(hosts, hosts_per_group)
runner.initiate()
reactor.run()





More information about the Twisted-Python mailing list