making sqlalchemy work with twisted (was Re: [Twisted-Python]SQLAbstraction Layer_

Jean-Paul Calderone exarkun at divmod.com
Wed Jan 18 22:32:28 EST 2006


On Wed, 18 Jan 2006 21:15:28 -0500, Paul G <paul-lists at perforge.com> wrote:
>
>----- Original Message ----- From: "Jean-Paul Calderone" 
><exarkun at divmod.com>
>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>Sent: Wednesday, January 18, 2006 7:55 PM
>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted- 
>Python]SQLAbstraction Layer_
>>On Wed, 18 Jan 2006 19:31:20 -0500, Paul G <paul-lists at perforge.com> wrote:
>>>
>>>----- Original Message ----- From: "Jean-Paul Calderone" 
>>><exarkun at divmod.com>
>>>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>>>Sent: Wednesday, January 18, 2006 7:10 PM
>>>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted- 
>>>Python]SQL Abstraction Layer_
>>>>On Wed, 18 Jan 2006 18:25:17 -0500, Paul G <paul-lists at perforge.com> 
>>>>wrote:
>>>>>
>>>>>to me, integrating sqlalchemy into twisted would ideally work in a way 
>>>>>where all sqlalchemy api access is async. as i stated in my original 
>>>>>mail, i currently believe that this could be possible to achieve by 
>>>>>making all of sqlalchemy's calls into the dbapi module async with 
>>>>>deferToThread(). if one does this, and it doesn't break something arcane 
>>>>>in sqlalchemy, we shouldn't have to worry about deferreds in the client 
>>>>>code. is there a reason why this wouldn't work or why it shouldn't be 
>>>>>done that i am missing?
>>>>
>>>>I think you are missing the fact that if you do this, attribute access 
>>>>will result in a Deferred, not the value of the attribute from the 
>>>>database, which is not available yet.
>>>>So client code will have to deal with Deferreds, and in an extremely 
>>>>unusual manner - every attribute lookup will return a new Deferred.
>>>
>>>this would be very much like the 'future' in the actor-based concurrency 
>>>model, so nothing terribly unusual. however, no, this is not what i want 
>>>to do. maybe my (bad) ascii art will help:
>>>
>>>normal: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine -> 
>>>synchronous db query to dbapi -> return to sqlengine -> return to accessor 
>>>-> return attribute
>>>
>>>new: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine -> 
>>>async call into dbapi with deferToThread-> control returned to reactor -> 
>>>another coop thread gets control
>>>
>>>... async dbapi result handler ->  return to sqlengine -> return to 
>>>accessor -> return attribute
>>>
>>>did i explain what i mean well?
>>
>>Yes.  Unfortunately, this cannot be implemented in CPython without going to 
>>extreme lengths.
>
>it just occurred to me what i was missing. namely, while the dbapi wrapper 
>can deferToThread() for calls into the real dbapi module, there's no 
>good/obvious way to yield control to the reactor and have the deferred's 
>callback return execution to the callsite which invoked the dbapi wrapper in 
>the first place. is this what you are alluding to?
>
>if so, i suspect there might be some generator magic which can make this 
>work. is this what you meant by 'extreme lengths'?

No, generators are easy.  Extreme lengths are extension modules that move pieces of the C call stack around.  See Stackless Python and the greenlets module.

>>  Also, there is some discussion among the core Twisted developers whether 
>>it even represents a good idea at all.  I think the split is currently 
>>something like 4 to 1 against.  If you look in either my blog or glyph's 
>>blog for "concurrency" you will find some exposition on the matter.
>
>there are indeed very good arguments for not doing this in the general case. 
>i believe the decision to make pb users be aware that the objects are remote 
>follows from that. i've found that this makes using pb easier and apps using 
>it easier to design (from a performance risk perspective) and easier to 
>debug, as well as more robust. i wouldn't argue for making attribute access 
>implicitly do asynchronous things in the general case, or in most cases 
>even.
>
>however, in the sqlalchemy case, at least for my purposes, it would make 
>sense to do it to make the integration more seamless.
>

You may have more reasons to claim this than you have presented, but what you have presented reduces to "it is a good idea in this case because I think it is a good idea".

I still think it's a bad idea :)

Jean-Paul




More information about the Twisted-Python mailing list