[Twisted-Python] Using sqlalchemy in twisted.

Peter Cai newptcai at gmail.com
Thu Mar 5 00:17:45 EST 2009

I'm not quite sure, but I think I'm pretty careful of sharing objects
between threads.
1st, I only cached as few as possible orm objects.  I tried to detach them,
but I found that if I detach them,  I can't access any of their fields any

2nd, I create new orm objects based on client request, pass them to class
Database and then merge them to scoped sessions, change, commit and then
discard these objects.

3rd, I switch to sqlite frequently to check if there is any database
operation outside Database, because sqlite doesn't allow multi-thread

Actually it seems to work until 2 or 3 days ago suddenly cases hang the

Ah, as I've already written lots of code in ORM, I think maybe I should try
to change Database to use a dedicated thread to handle all database

That might be a bottle neck of my application, but I really can't give up
orm as these mapper classes are used everywhere in my application.

On Thu, Mar 5, 2009 at 3:04 AM, Valentino Volonghi <dialtone at gmail.com>wrote:

> Hash: SHA1
> On Mar 4, 2009, at 3:28 AM, Peter Cai wrote:
>  The code doesn't work.   When I limit the thread numbers to 1
>>   reactor.suggestThreadPoolSize(1)
>> Everything goes fine.  Other wise the server would be blocked and must
>> be killed by "kill 9 ...".
>> The result conflicts with my understanding of sqlalchemy.  Since I
>> don't share any object between threads, there should be no problem!
> I'm pretty sure you can't say this with full certainty and actually you are
> just wrong based on the code you showed. deferToThread doesn't use
> the same thread every time you call it, there's absolutely no guarantee
> in that and sqlalchemy keeps state around in that thread related to that
> object it returned. If you want to do any operations you need either to
> detach the object from the session before returning it and do any
> modification
> on the same object in another thread after reattaching it (pretty
> cumbersome).
> Or write your own threadpool that gives names to threads so that you can
> have a guarantee that you always call the same thread when working with
> a bunch of objects.
> Nonetheless though sqlalchemy is a huge project and I'm pretty sure it has
> some deadlocks and race conditions around which means that even taking
> care of these issues you'll have other problems when dealing with the orm.
> If you want to use sqlalchemy don't use its orm but just the query builder,
> it's the only sane way to integrate with twisted. Which anyway IMHO is the
> best way to use it anyway because it uses a lot less memory since it
> doesn't
> have to always cache objects because you control everything and can make
> that call when you really need it.
>  Ah....  It always have risk to use something you haven't tried
>> before ....
>> I think I have no choice but always set thread pool size to 1 ...
> Not entirely true.
> - --
> Valentino Volonghi aka Dialtone
> Now running MacOS X 10.5
> Home Page: http://www.twisted.it
> http://www.adroll.com
> Version: GnuPG v1.4.9 (Darwin)
> iEYEARECAAYFAkmu0NoACgkQ9Llz28widGXBawCg32svBsLqsIRLzvzOThgR4sA0
> 5UkAoIgNfyUDPl9c0nwSud0sem3aKjz5
> =2XIX
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

look to the things around you,the immediate world around you, if you are
alive,it will mean something to you ——Paul Strand
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090305/9a952ddc/attachment.htm 

More information about the Twisted-Python mailing list