making sqlalchemy work with twisted (was Re: [Twisted-Python]SQLAbstraction Layer_

Paul G paul-lists at perforge.com
Wed Jan 18 21:15:28 EST 2006


----- Original Message ----- 
From: "Jean-Paul Calderone" <exarkun at divmod.com>
To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
Sent: Wednesday, January 18, 2006 7:55 PM
Subject: Re: making sqlalchemy work with twisted (was Re: 
[Twisted-Python]SQLAbstraction Layer_


> On Wed, 18 Jan 2006 19:31:20 -0500, Paul G <paul-lists at perforge.com> 
> wrote:
>>
>>----- Original Message ----- From: "Jean-Paul Calderone" 
>><exarkun at divmod.com>
>>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>>Sent: Wednesday, January 18, 2006 7:10 PM
>>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted- 
>>Python]SQL Abstraction Layer_
>>>On Wed, 18 Jan 2006 18:25:17 -0500, Paul G <paul-lists at perforge.com> 
>>>wrote:
>>>>
>>>>to me, integrating sqlalchemy into twisted would ideally work in a way 
>>>>where all sqlalchemy api access is async. as i stated in my original 
>>>>mail, i currently believe that this could be possible to achieve by 
>>>>making all of sqlalchemy's calls into the dbapi module async with 
>>>>deferToThread(). if one does this, and it doesn't break something arcane 
>>>>in sqlalchemy, we shouldn't have to worry about deferreds in the client 
>>>>code. is there a reason why this wouldn't work or why it shouldn't be 
>>>>done that i am missing?
>>>
>>>I think you are missing the fact that if you do this, attribute access 
>>>will result in a Deferred, not the value of the attribute from the 
>>>database, which is not available yet.
>>>So client code will have to deal with Deferreds, and in an extremely 
>>>unusual manner - every attribute lookup will return a new Deferred.
>>
>>this would be very much like the 'future' in the actor-based concurrency 
>>model, so nothing terribly unusual. however, no, this is not what i want 
>>to do. maybe my (bad) ascii art will help:
>>
>>normal: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine -> 
>>synchronous db query to dbapi -> return to sqlengine -> return to 
>>accessor -> return attribute
>>
>>new: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine -> 
>>async call into dbapi with deferToThread-> control returned to reactor -> 
>>another coop thread gets control
>>
>>... async dbapi result handler ->  return to sqlengine -> return to 
>>accessor -> return attribute
>>
>>did i explain what i mean well?
>
> Yes.  Unfortunately, this cannot be implemented in CPython without going 
> to extreme lengths.

it just occurred to me what i was missing. namely, while the dbapi wrapper 
can deferToThread() for calls into the real dbapi module, there's no 
good/obvious way to yield control to the reactor and have the deferred's 
callback return execution to the callsite which invoked the dbapi wrapper in 
the first place. is this what you are alluding to?

if so, i suspect there might be some generator magic which can make this 
work. is this what you meant by 'extreme lengths'?

>  Also, there is some discussion among the core Twisted developers whether 
> it even represents a good idea at all.  I think the split is currently 
> something like 4 to 1 against.  If you look in either my blog or glyph's 
> blog for "concurrency" you will find some exposition on the matter.

there are indeed very good arguments for not doing this in the general case. 
i believe the decision to make pb users be aware that the objects are remote 
follows from that. i've found that this makes using pb easier and apps using 
it easier to design (from a performance risk perspective) and easier to 
debug, as well as more robust. i wouldn't argue for making attribute access 
implicitly do asynchronous things in the general case, or in most cases 
even.

however, in the sqlalchemy case, at least for my purposes, it would make 
sense to do it to make the integration more seamless.

-p 





More information about the Twisted-Python mailing list