making sqlalchemy work with twisted (was Re: [Twisted-Python]SQLAbstraction Layer_
exarkun at divmod.com
Wed Jan 18 22:32:28 EST 2006
On Wed, 18 Jan 2006 21:15:28 -0500, Paul G <paul-lists at perforge.com> wrote:
>----- Original Message ----- From: "Jean-Paul Calderone"
><exarkun at divmod.com>
>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>Sent: Wednesday, January 18, 2006 7:55 PM
>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted-
>>On Wed, 18 Jan 2006 19:31:20 -0500, Paul G <paul-lists at perforge.com> wrote:
>>>----- Original Message ----- From: "Jean-Paul Calderone"
>>><exarkun at divmod.com>
>>>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>>>Sent: Wednesday, January 18, 2006 7:10 PM
>>>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted-
>>>Python]SQL Abstraction Layer_
>>>>On Wed, 18 Jan 2006 18:25:17 -0500, Paul G <paul-lists at perforge.com>
>>>>>to me, integrating sqlalchemy into twisted would ideally work in a way
>>>>>where all sqlalchemy api access is async. as i stated in my original
>>>>>mail, i currently believe that this could be possible to achieve by
>>>>>making all of sqlalchemy's calls into the dbapi module async with
>>>>>deferToThread(). if one does this, and it doesn't break something arcane
>>>>>in sqlalchemy, we shouldn't have to worry about deferreds in the client
>>>>>code. is there a reason why this wouldn't work or why it shouldn't be
>>>>>done that i am missing?
>>>>I think you are missing the fact that if you do this, attribute access
>>>>will result in a Deferred, not the value of the attribute from the
>>>>database, which is not available yet.
>>>>So client code will have to deal with Deferreds, and in an extremely
>>>>unusual manner - every attribute lookup will return a new Deferred.
>>>this would be very much like the 'future' in the actor-based concurrency
>>>model, so nothing terribly unusual. however, no, this is not what i want
>>>to do. maybe my (bad) ascii art will help:
>>>normal: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine ->
>>>synchronous db query to dbapi -> return to sqlengine -> return to accessor
>>>-> return attribute
>>>new: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine ->
>>>async call into dbapi with deferToThread-> control returned to reactor ->
>>>another coop thread gets control
>>>... async dbapi result handler -> return to sqlengine -> return to
>>>accessor -> return attribute
>>>did i explain what i mean well?
>>Yes. Unfortunately, this cannot be implemented in CPython without going to
>it just occurred to me what i was missing. namely, while the dbapi wrapper
>can deferToThread() for calls into the real dbapi module, there's no
>good/obvious way to yield control to the reactor and have the deferred's
>callback return execution to the callsite which invoked the dbapi wrapper in
>the first place. is this what you are alluding to?
>if so, i suspect there might be some generator magic which can make this
>work. is this what you meant by 'extreme lengths'?
No, generators are easy. Extreme lengths are extension modules that move pieces of the C call stack around. See Stackless Python and the greenlets module.
>> Also, there is some discussion among the core Twisted developers whether
>>it even represents a good idea at all. I think the split is currently
>>something like 4 to 1 against. If you look in either my blog or glyph's
>>blog for "concurrency" you will find some exposition on the matter.
>there are indeed very good arguments for not doing this in the general case.
>i believe the decision to make pb users be aware that the objects are remote
>follows from that. i've found that this makes using pb easier and apps using
>it easier to design (from a performance risk perspective) and easier to
>debug, as well as more robust. i wouldn't argue for making attribute access
>implicitly do asynchronous things in the general case, or in most cases
>however, in the sqlalchemy case, at least for my purposes, it would make
>sense to do it to make the integration more seamless.
You may have more reasons to claim this than you have presented, but what you have presented reduces to "it is a good idea in this case because I think it is a good idea".
I still think it's a bad idea :)
More information about the Twisted-Python