making sqlalchemy work with twisted (was Re: [Twisted-Python] SQL Abstraction Layer_

[Paul G]

----- Original Message ----- 
From: "Valentino Volonghi aka Dialtone" <dialtone at divmod.com>
To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
Sent: Wednesday, January 18, 2006 5:28 AM
Subject: Re: [Twisted-Python] SQL Abstraction Layer


On Tue, Jan 17, 2006 at 09:34:11PM -0500, Paul G wrote:
>> correct me if i'm wrong, but hacking/monkeypatching the sqlalchemy
>> accessors which implement this lazy load when attributes are accessed to
>> run their queries with deferToThread() would remove this problem, would 
>> it
>> not? more generally speaking, if you're not going to do explicit
>> deferToThread()s in your code (and avoid all attribute access), but 
>> rather,
>> say, insert them into the appropriate point in sqlalchemy (sqlengine 
>> seems
>> ro tbe the right place, or just a wrapper around the dbapi module), would
>> this not handle the issue as well? am i asking the wrong person/on the
>> wrong list? ;]

>I don't think it's the right approach to patch sqlalchemy to run attribute
>access in a thread. You would still get a deferred from an attribute access
>and this is pretty hugly to handle.

i'm not sure we're on the same page here. alternatively, i don't agree with 
your reasoning. if we use eager mode, objects get created when we call the 
mapper. in lazy mode, they can appear as a result of attribute access. to 
me, the fact that the two are different are a language syntax issue - 
semantically, i see both as a message to a slot on an object.  this is, 
iirc, how python allows sqlalchemy to intercept with attribute accessors as 
well ;)

to me, integrating sqlalchemy into twisted would ideally work in a way where 
all sqlalchemy api access is async. as i stated in my original mail, i 
currently believe that this could be possible to achieve by making all of 
sqlalchemy's calls into the dbapi module async with deferToThread(). if one 
does this, and it doesn't break something arcane in sqlalchemy, we shouldn't 
have to worry about deferreds in the client code. is there a reason why this 
wouldn't work or why it shouldn't be done that i am missing?

you could advance the argument that not having a deferToThread() in the 
client code would obscure the fact that the current 'cooperative thread' is 
yielding execution at that point, but that can be elucidated in the comments 
should one have a need to do that, while keeping the code much cleaner.

> Just using the eager mode would be perfect, the reason why I don't believe
> that much in lazy queries is that you will end up using all the 
> information,
> otherwise it's better to query without using the ORM.

there are plenty of examples where lazy loading is useful in an ORM context 
and where you wouldn't want to do your own queries. in fact, i don't see why 
wanting to control loading behaviour would make the advantages of an ORM 
suddenly go away - to me, the principal advantage is making the code cleaner 
and, in the common case, masking the stuff that's going on under the hood. 
this becomes even more important, imho, when you want to lazy load things - 
not less.

>Currently SQLAlchemy is so good because it doesn't try to be clever but 
>rather
>only exposes the primitives and tries hard to be thread safe so that people
>will be free to use it in the best way they want.

full agreement here. i love the way it's designed - simple, gives you access 
to the various levels of functionality independent of the others and leaves 
you free to use it however works best for you at any given point in 
time/code. it's one of the best ORMs, design-wise, that i've seen; the only 
thing that's missing is some docs for the UnitOfWork stuff.

-p