[Twisted-Python] bsddb?

[Bill la Forge]

But it blocks, mm?
 
I could break things into small chuncks, so I could likely get away with
not using a thread. But shouldn't it ideally be accessed from a thread?
 
I'll note that you can open bsddb (using the new interface, not the
one documented in Python) in a thread-safe way, but any given iteration
(using a cursor object) must be done from a single thread--cursors
are not thread-free.
 
One approach might be to create a cursor in a server thread and then
use a fixed-size queue to retrieve the iterations, but then you'll need
at least as many server threads as you have open cursors.
 
So the best approach might be to have a cursor manager which handles
many such cursors/queues running on a single thread. (Invoking the
manager, it returns a master queue for handling subsequent requests.)
But then it sounds like a dedicated thread is again the best approach.
 
It seems to me that a nice general solution would be a project in its own 
right. (I've found bsddb to be pretty powerful/useful/fast/handy.)
 
Ah, Atop looks interesting. Thanks!
 
Bill

Matt Goodall <matt at pollenation.net> wrote:
On Fri, 2004-05-14 at 08:17, Bill la Forge wrote:
> New to twisted, interested in porting a project, but I can't find
> anything on using bsddb.

Open a bdb at the start of the app, use it as necessary, close the bdb
at the end of the app. What could be easier ;-).

It's probably worth creating a Service
(http://twistedmatrix.com/documents/current/api/) to help manage the
bsddb resource.

> I assume I need a dedicated thread. Has anyone else worked on this?

You probably only need a thread for operations that you know will take a
long time. Retrieving a single, keyed object should be so quick with
bsdbd that it's probably not with the effort/expense of a thread. On the
other hand, iterating an entire bdb will (possibly) take quite some
time, so a thread may be a good idea. (Note that Python's bsdbd is not
threadsafe so you will have to manage that yourself.)

I don't know enough about the bsddb api but I suspect it's possible to
avoid threads altogether by iterating an entire bdb in small chunks and
relinquishing control to the event loop at the end of each chunk to
allow other events to run.

> (I did a web search, but couldn't relate what I found to any code in
> the 1.2 release I just downloaded. I also read the May list archive.)

You may find Atop (http://www.divmod.org/Home/Projects/Atop/index.html)
interesting.

Hope this helps.

Cheers, Matt

-- 
__
/ \__ Matt Goodall, Pollenation Internet Ltd
\__/ \ w: http://www.pollenation.net
__/ \__/ e: matt at pollenation.net
/ \__/ \ t: +44 (0)113 2252500
\__/ \__/
/ \ Any views expressed are my own and do not necessarily
\__/ reflect the views of my employer.


_______________________________________________
Twisted-Python mailing list
Twisted-Python at twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Bill la Forge
http://www.geocities.com/laforge49/
Yahoo! India Matrimony: Find your partner online.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20040514/63bc01b3/attachment.htm