[Twisted-Python] Re: Should I use asynchronous programming in my own modules?

Thu Oct 18 13:49:21 EDT 2007

Jürgen Strass wrote:

> My idea now is that depending on the number of child elements, looping
> could take some time. So instead I'd use twisted's reactor, specifically
> its callLater method like this (it's only pseudo code!):
> 
> class Generator:
> 
>    def generate_html( self ):
>       self.d = defer.Deferred()
>       self.startProcessing()
>       return self.d
> 
>    def startProcessing( self ):
>       self.current_element = root_elem
>       self.processNextElement()
> 
>    def processNextElement( self ):
>       if more elements to process:
>          if current_element.type = chapter
>             reactor.callLater( 0, processChapter, current_element )
>       .....
>       else:
>          d.callback( "finished" )

This is not so good. You have taken away your users option to control which
thread this processing is performed in, because it has to run in the
reactor's thread to avoid breaking Deferred's threading rules. Also your
users are not getting any benefit from the incremental nature of this code.
For example they cant get access to the first chunk of html until after
callback("finished"). At least, not without more complexity. They cant
decide to stop processing early because they were only interested in the
html <head>.

> It's more or less clear to me how to divide the traversal of such a dom
> tree into discrete steps

It seems like you are confusing this goal - the ability to perform work in
incremental steps - with the use of twisted's reactor to schedule those
steps.

Set twisted aside for a moment. I propose one good pythonic interface to
your html creation code may be a generate_html_iter() method, which returns
an iterator over the documents html fragments. You can implement this using
the processNextElement approach you suggested, although python generator
functions may be more convenient.

A thin wrapper around this iterator could use reactor.callLater to schedule
it, then fire a callback when complete. An equally thin wrapper could use
PostMessage to calculate the document in the background of a win32 gui. Or
it could feed a pull producer.  Alternatively it could be run in another
thread with deferToThread(lambda:''.join(g.generate_html_iter()))

I hope this helps,