[Twisted-Python] Should I use asynchronous programming in my own modules?

Jürgen Strass jrg718 at gmx.net
Thu Oct 18 14:04:13 EDT 2007

Christopher Armstrong wrote:
> You don't need to make the module depend on Twisted, but you also
> don't need to force users to use a thread. Just make sure the library
> knows how to parse and process incrementally; then your asynchronous
> users can pass in chunks of data as they receive them and your other
> users can pass in everything at once. Basically, it's a matter of
> inverting your library's loop that would otherwise go "read; process;"
> to "when process_more_data is called, process that data".
As I already explained in my replies to Jean-Paul Calderone and Itamar 
Shtull-Trauring, I'm not sure if I have fully understood how this works 
in particular. The example of a parser probably wasn't well chosen, 
because in all replies to my original posting people seem to assume that 
I need to do something in response to an external event (e.g. I/O 
events). Please, better think of a long running CPU bound algorithm 
perhaps. From twisted, I'd like to start that algorithm in the 
background and get notified when it has finished. It shouldn't ever 
block the application for too long. In other applications, I'd like to 
call the algorithm in a synchronous way.

It is most likely that my difficulty in understanding results from the 
fact that in such a case there is no external event that could trigger 
the processing of the individual steps the algorithm has. The best I've 
come up with so far uses a wrapper around a synchronous interface that 
provides methods for calling the next step of the algorithm and for 
testing if the algorithm has finished. What I'd like to know is if this 
solution basically is the style one has to use when writing algorithms 
divided into chunks and if the approach to integrate it into twisted is 
well chosen. Also, I don't fully understand the implications this 
programming style has on long running loops. I have the impression that 
I would need to divide loops into several methods: loopInit, 
loopCondition, loopNextStep and loopFinished. Is this correct?

Following is my code for a counter class which also uses the wrapper 
class I've described above. Using the wrapper class, it is possibly to 
run several counters concurrently. The Counter class is already 
decoupled from twisted. In case I want to use the synchronous version, 
I'd use Counter, in case I want to use the asynchronous version with 
deferreds, I'd use TwistedCounter.


from twisted.internet import reactor, defer

class Counter:
   def __init__( self, id, limit ):
      self.id = id
      self.limit = limit
      self.count = 0
   def limitNotReached( self ):
      return self.count < self.limit
   def increment( self ):
      if self.limitNotReached():
         print "Counter %d: %d" % ( self.id, self.count )
         self.count = self.count + 1
   def run( self ):
      while self.limitNotReached():
class TwistedCounter:
   def __init__( self, counter ):
      self.counter = counter
   def run( self ):
      self.d = defer.Deferred()
      return self.d
   def increment( self ):
      if self.counter.limitNotReached():
         reactor.callLater( 0, self.increment )
         self.d.callback( "finished" )
def counters_finished( result ):
   print "  traditional counters finished."

print "-- Run traditional counter:\n"

traditional_c = Counter( 1, 130 )

print "\n-- Run twisted counters concurrently:\n"

c1 = TwistedCounter( Counter( 2, 100 ) )
c2 = TwistedCounter( Counter( 3, 150 ) )
d1 = c1.run()
d2 = c2.run()
d = defer.DeferredList( [ d1, d2 ] )
d.addCallback( counters_finished )



What I can't fully imagine is if I can use this technique in all 
possible cases. In case I need to nest calls or algorithms, I'll run 
into the problem of not being able to call any of the asynchronous 
wrappers from the synchronous parts of my library, because those 
wrappers return deferreds which are part of twisted. So I'd end up with 
two versions of my library again.


More information about the Twisted-Python mailing list