making sqlalchemy work with twisted (was Re: [Twisted-Python]SQL Abstraction Layer_
exarkun at divmod.com
Wed Jan 18 19:55:32 EST 2006
On Wed, 18 Jan 2006 19:31:20 -0500, Paul G <paul-lists at perforge.com> wrote:
>----- Original Message ----- From: "Jean-Paul Calderone"
><exarkun at divmod.com>
>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>Sent: Wednesday, January 18, 2006 7:10 PM
>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted-
>Python]SQL Abstraction Layer_
>>On Wed, 18 Jan 2006 18:25:17 -0500, Paul G <paul-lists at perforge.com> wrote:
>>>to me, integrating sqlalchemy into twisted would ideally work in a way
>>>where all sqlalchemy api access is async. as i stated in my original mail,
>>>i currently believe that this could be possible to achieve by making all
>>>of sqlalchemy's calls into the dbapi module async with deferToThread(). if
>>>one does this, and it doesn't break something arcane in sqlalchemy, we
>>>shouldn't have to worry about deferreds in the client code. is there a
>>>reason why this wouldn't work or why it shouldn't be done that i am
>>I think you are missing the fact that if you do this, attribute access will
>>result in a Deferred, not the value of the attribute from the database,
>>which is not available yet.
>>So client code will have to deal with Deferreds, and in an extremely
>>unusual manner - every attribute lookup will return a new Deferred.
>this would be very much like the 'future' in the actor-based concurrency
>model, so nothing terribly unusual. however, no, this is not what i want to
>do. maybe my (bad) ascii art will help:
>normal: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine ->
>synchronous db query to dbapi -> return to sqlengine -> return to accessor
>-> return attribute
>new: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine ->
>async call into dbapi with deferToThread-> control returned to reactor ->
>another coop thread gets control
>... async dbapi result handler -> return to sqlengine -> return to accessor
>-> return attribute
>did i explain what i mean well?
Yes. Unfortunately, this cannot be implemented in CPython without going to extreme lengths. Also, there is some discussion among the core Twisted developers whether it even represents a good idea at all. I think the split is currently something like 4 to 1 against. If you look in either my blog or glyph's blog for "concurrency" you will find some exposition on the matter.
More information about the Twisted-Python