[Twisted-Python] Re: So how does everyone use an ORM in twisted?

[Ed Suominen]

For what it's worth, I've never used SQLAlchemy's ORM capabilities and 
don't support it in sAsync. I agree that it really isn't compatible with 
the asynchronous nature of Twisted, and I don't see it as adding 
anything. To me, the power of SQLAlchemy is in the clean and convenient 
way it permits you to access the underlying relational database, and 
sAsync merely lets you do that access in an asynchronous manner, in a 
single thread or with multiple threads.

Best regards, Ed

David Bolen wrote:
> Don Dwiggins <ddwiggins at advpubtech.com> writes:
> 
>> So if someone wanted the use of the ORM, would you say that they
>> should access the objects through deferreds, just as they would with
>> SQL?  Or is it worse than that?
> 
> If by deferreds you mean pushing the ORM operation (including all
> possible ancillary object loading) into a separate thread via
> deferToThread, yes, that's a possibility.  But that likely will only
> work best if you can work with a structure where the results of
> database operations can be encapsulated into a fairly simple
> interface, and not as much if the mainline code wants access to the
> entire object tree.  Otherwise, you end up doing a lot of work to
> protect against the very features (especially lazy loading) that often
> makes an ORM attractive to people in order to ensure that the object
> tree is safe to use from within the non-ORM thread.
> 
> If, however, you mean some other use of deferreds, then no - I agree
> with Jean-Paul's earlier comment that most ORMs are simply not
> designed to play nicely with the non-blocking Twisted main loop.
> Database operations typically trigger from simple attribute access, or
> from within, say an iteration loop.  They then block, so unless they
> happen to execute very quickly you're blocking the Twisted main loop.
> And it's tough to insert a normal deferred callback into that
> equation.
> 
> You can try to offload the underlying DB operation to a thread pool,
> but then you start having to fight to explicitly control all DB
> access, some of which can occur during a simple attribute reference -
> at which point there's no reasonable way to use a Deferred to postpone
> the attribute lookup.
> 
> It's been a while since I looked at it, so I'm not sure of its status,
> but sAsync was actually a interesting effort to at least get as good a
> wrapper as possible, but it doesn't preclude having to be cautious
> about ensuring any objects returned from the wrapper have been fully
> loaded, or that any interaction that might require going back to the
> database only be through wrapped methods and not direct attribute
> access.  This can turn out to be a bunch of extra work when you start
> talking about walking lists of related (one to many) records,
> traversing the links of foreign keys among records, etc.. much of
> which is the attractive part of ORMs for those who like them.
> 
> I suppose it is only fair to also say that if you don't mind some
> arbitrary delays in the mainline code, there's nothing stopping you
> from just using an ORM like SQLAlchemy right inline and ignoring the
> whole issue.  For smaller databases or ones whose server are on the
> local host (or internal to the process like sqlite), it might not even
> be a real problem.  It's certainly flawed structurally and likely
> won't scale, but that doesn't mean it might not be suitable in certain
> cases if you know the tradeoffs.
> 
> -- David
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> 
>