[Twisted-Python] File persistence without blocking the main event loop?

[Jean-Paul Calderone]

On Mon, 18 May 2009 08:31:56 -0400, Itamar Shtull-Trauring <itamar at itamarst.org> wrote:
>On Mon, 2009-05-18 at 19:34 +0800, Arun Thampi wrote:
>> Hi guys - I'm just beginning to learn about Twisted and was wondering
>> if there is a way to use the built-in Twisted persistence libraries
>> such as dirdbm etc. without blocking the main event loop? I read about
>> an approach which uses deferToThread (comment#3
>> here: http://code.activestate.com/recipes/413609/) but if you are
>> persisting quite often, doesn't the creation of threads on every
>> persist-call make it very expensive?
>
>deferToThread uses a threadpool, so it's not *very* slow. It does slow
>things down a bit, so it's better to do chunks of work rather than doing
>writes separately. That being said, writing to disk in main eventloop is
>often fast enough in practice, depending on how strong your latency
>guarantees need to be.
>

More importantly, don't use dirdbm.  It's not a reliable data storage
library.  In fact, you probably shouldn't use anything in twisted.persisted.

You might be interested in <http://www.divmod.org/trac/wiki/DivmodAxiom>.

Jean-Paul