making sqlalchemy work with twisted (was Re:[Twisted-Python]SQLAbstraction Layer_

Paul G paul-lists at perforge.com
Wed Jan 18 22:55:43 EST 2006


----- Original Message ----- 
From: "Jean-Paul Calderone" <exarkun at divmod.com>
To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
Sent: Wednesday, January 18, 2006 10:32 PM
Subject: Re: making sqlalchemy work with twisted (was 
Re:[Twisted-Python]SQLAbstraction Layer_


> On Wed, 18 Jan 2006 21:15:28 -0500, Paul G <paul-lists at perforge.com> 
> wrote:
>>
>>----- Original Message ----- From: "Jean-Paul Calderone" 
>><exarkun at divmod.com>
>>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>>Sent: Wednesday, January 18, 2006 7:55 PM
>>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted- 
>>Python]SQLAbstraction Layer_
>>>On Wed, 18 Jan 2006 19:31:20 -0500, Paul G <paul-lists at perforge.com> 
>>>wrote:
>>>>
>>>>----- Original Message ----- From: "Jean-Paul Calderone" 
>>>><exarkun at divmod.com>
>>>>To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>>>>Sent: Wednesday, January 18, 2006 7:10 PM
>>>>Subject: Re: making sqlalchemy work with twisted (was Re: [Twisted- 
>>>>Python]SQL Abstraction Layer_
>>>>>On Wed, 18 Jan 2006 18:25:17 -0500, Paul G <paul-lists at perforge.com> 
>>>>>wrote:
>>>>>>
>>>>>>to me, integrating sqlalchemy into twisted would ideally work in a way 
>>>>>>where all sqlalchemy api access is async. as i stated in my original 
>>>>>>mail, i currently believe that this could be possible to achieve by 
>>>>>>making all of sqlalchemy's calls into the dbapi module async with 
>>>>>>deferToThread(). if one does this, and it doesn't break something 
>>>>>>arcane in sqlalchemy, we shouldn't have to worry about deferreds in 
>>>>>>the client code. is there a reason why this wouldn't work or why it 
>>>>>>shouldn't be done that i am missing?
>>>>>
>>>>>I think you are missing the fact that if you do this, attribute access 
>>>>>will result in a Deferred, not the value of the attribute from the 
>>>>>database, which is not available yet.
>>>>>So client code will have to deal with Deferreds, and in an extremely 
>>>>>unusual manner - every attribute lookup will return a new Deferred.
>>>>
>>>>this would be very much like the 'future' in the actor-based concurrency 
>>>>model, so nothing terribly unusual. however, no, this is not what i want 
>>>>to do. maybe my (bad) ascii art will help:
>>>>
>>>>normal: attribute access -> sqlalchemy accessor -> sqlalchemy 
>>>>sqlengine -> synchronous db query to dbapi -> return to sqlengine -> 
>>>>return to accessor -> return attribute
>>>>
>>>>new: attribute access -> sqlalchemy accessor -> sqlalchemy sqlengine -> 
>>>>async call into dbapi with deferToThread-> control returned to 
>>>>reactor -> another coop thread gets control
>>>>
>>>>... async dbapi result handler ->  return to sqlengine -> return to 
>>>>accessor -> return attribute
>>>>
>>>>did i explain what i mean well?
>>>
>>>Yes.  Unfortunately, this cannot be implemented in CPython without going 
>>>to extreme lengths.
>>
>>it just occurred to me what i was missing. namely, while the dbapi wrapper 
>>can deferToThread() for calls into the real dbapi module, there's no 
>>good/obvious way to yield control to the reactor and have the deferred's 
>>callback return execution to the callsite which invoked the dbapi wrapper 
>>in the first place. is this what you are alluding to?
>>
>>if so, i suspect there might be some generator magic which can make this 
>>work. is this what you meant by 'extreme lengths'?
>
> No, generators are easy.  Extreme lengths are extension modules that move 
> pieces of the C call stack around.  See Stackless Python and the > 
> greenlets module.

yeah, i've looked at those in the past. are you saying they are the only way 
to do this and generators won't work?

>
>>>  Also, there is some discussion among the core Twisted developers 
>>> whether it even represents a good idea at all.  I think the split is 
>>> currently something like 4 to 1 against.  If you look in either my blog 
>>> or glyph's blog for "concurrency" you will find some exposition on the 
>>> matter.
>>
>>there are indeed very good arguments for not doing this in the general 
>>case. i believe the decision to make pb users be aware that the objects 
>>are remote follows from that. i've found that this makes using pb easier 
>>and apps using it easier to design (from a performance risk perspective) 
>>and easier to debug, as well as more robust. i wouldn't argue for making 
>>attribute access implicitly do asynchronous things in the general case, or 
>>in most cases even.
>>
>>however, in the sqlalchemy case, at least for my purposes, it would make 
>>sense to do it to make the integration more seamless.
>>
>
> You may have more reasons to claim this than you have presented, but what 
> you have presented reduces to "it is a good idea in this case > because I 
> think it is a good idea".

nail. head. ;] in short, i trust myself to know that in those cases, an 
async call happens. this wouldn't be good in a general use framework, but 
for an internal product, the cleanliness of the code (and ability to do lazy 
loading) tip the scale, imo.

so, outside of greenlets (reportedly, they make weird things happen from 
time to time, which isn't acceptable) and stackless (sexy, but 
unsupported/not actively developed/does it still work/will it stop 
working?), is there a way to get this done?

-p 





More information about the Twisted-Python mailing list