making sqlalchemy work with twisted (was Re: [Twisted-Python]SQL Abstraction Layer_

Jean-Paul Calderone exarkun at divmod.com
Wed Jan 18 19:55:32 EST 2006


On Wed, 18 Jan 2006 19:31:20 -0500, Paul G <paul-lists at perforge.com> wrote:
>
>----- Original Message ----- From: "Jean-Paul Calderone" 
><exarkun at divmod.com>
>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>Sent: Wednesday, January 18, 2006 7:10 PM
>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted- 
>Python]SQL Abstraction Layer_
>>On Wed, 18 Jan 2006 18:25:17 -0500, Paul G <paul-lists at perforge.com> wrote:
>>>
>>>to me, integrating sqlalchemy into twisted would ideally work in a way 
>>>where all sqlalchemy api access is async. as i stated in my original mail, 
>>>i currently believe that this could be possible to achieve by making all 
>>>of sqlalchemy's calls into the dbapi module async with deferToThread(). if 
>>>one does this, and it doesn't break something arcane in sqlalchemy, we 
>>>shouldn't have to worry about deferreds in the client code. is there a 
>>>reason why this wouldn't work or why it shouldn't be done that i am 
>>>missing?
>>
>>I think you are missing the fact that if you do this, attribute access will 
>>result in a Deferred, not the value of the attribute from the database, 
>>which is not available yet.
>>So client code will have to deal with Deferreds, and in an extremely 
>>unusual manner - every attribute lookup will return a new Deferred.
>
>this would be very much like the 'future' in the actor-based concurrency 
>model, so nothing terribly unusual. however, no, this is not what i want to 
>do. maybe my (bad) ascii art will help:
>
>normal: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine -> 
>synchronous db query to dbapi -> return to sqlengine -> return to accessor 
>-> return attribute
>
>new: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine -> 
>async call into dbapi with deferToThread-> control returned to reactor -> 
>another coop thread gets control
>
>... async dbapi result handler ->  return to sqlengine -> return to accessor 
>-> return attribute
>
>did i explain what i mean well?

Yes.  Unfortunately, this cannot be implemented in CPython without going to extreme lengths.  Also, there is some discussion among the core Twisted developers whether it even represents a good idea at all.  I think the split is currently something like 4 to 1 against.  If you look in either my blog or glyph's blog for "concurrency" you will find some exposition on the matter.

Jean-Paul




More information about the Twisted-Python mailing list