[Twisted-Python] Should I use asynchronous programming in my own modules?

[Christopher Armstrong]

On 10/18/07, Jürgen Strass <jrg718 at gmx.net> wrote:

> To give some example, I'd like to develop my own simplified document
> format in XML and a corresponding parser.
...
> This module could be useful outside any twisted
> application, of course.
...
> The question is this:
> since parsing and generating large documents could block the reactor in
> a twisted app, should I use any of twisted's asynchronous programming
> features in this module (for better integration with twisted) or should
> I rather develop it in a traditional way and run it in a thread?

You don't need to make the module depend on Twisted, but you also
don't need to force users to use a thread. Just make sure the library
knows how to parse and process incrementally; then your asynchronous
users can pass in chunks of data as they receive them and your other
users can pass in everything at once. Basically, it's a matter of
inverting your library's loop that would otherwise go "read; process;"
to "when process_more_data is called, process that data".


> The last question is what criteria I could use to divide long lasting
> operations into chunks. In almost all books about asynchronous
> programming I only read that if they're too big, they could block the
> event loop. Of course, but how big is too big? And what's the measure
> for it? Milliseconds, number of operations, number of code lines - or
> what? Doesn't it depend entirely on the application at hand and how
> reactive it has to be?

"Bigness" here refers to time spent. And yes, it depends on the application.


-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/