[Twisted-Python] Re: So how does everyone use an ORM in twisted?

[David Bolen]

Don Dwiggins <ddwiggins at advpubtech.com> writes:

> So if someone wanted the use of the ORM, would you say that they
> should access the objects through deferreds, just as they would with
> SQL?  Or is it worse than that?

If by deferreds you mean pushing the ORM operation (including all
possible ancillary object loading) into a separate thread via
deferToThread, yes, that's a possibility.  But that likely will only
work best if you can work with a structure where the results of
database operations can be encapsulated into a fairly simple
interface, and not as much if the mainline code wants access to the
entire object tree.  Otherwise, you end up doing a lot of work to
protect against the very features (especially lazy loading) that often
makes an ORM attractive to people in order to ensure that the object
tree is safe to use from within the non-ORM thread.

If, however, you mean some other use of deferreds, then no - I agree
with Jean-Paul's earlier comment that most ORMs are simply not
designed to play nicely with the non-blocking Twisted main loop.
Database operations typically trigger from simple attribute access, or
from within, say an iteration loop.  They then block, so unless they
happen to execute very quickly you're blocking the Twisted main loop.
And it's tough to insert a normal deferred callback into that
equation.

You can try to offload the underlying DB operation to a thread pool,
but then you start having to fight to explicitly control all DB
access, some of which can occur during a simple attribute reference -
at which point there's no reasonable way to use a Deferred to postpone
the attribute lookup.

It's been a while since I looked at it, so I'm not sure of its status,
but sAsync was actually a interesting effort to at least get as good a
wrapper as possible, but it doesn't preclude having to be cautious
about ensuring any objects returned from the wrapper have been fully
loaded, or that any interaction that might require going back to the
database only be through wrapped methods and not direct attribute
access.  This can turn out to be a bunch of extra work when you start
talking about walking lists of related (one to many) records,
traversing the links of foreign keys among records, etc.. much of
which is the attractive part of ORMs for those who like them.

I suppose it is only fair to also say that if you don't mind some
arbitrary delays in the mainline code, there's nothing stopping you
from just using an ORM like SQLAlchemy right inline and ignoring the
whole issue.  For smaller databases or ones whose server are on the
local host (or internal to the process like sqlite), it might not even
be a real problem.  It's certainly flawed structurally and likely
won't scale, but that doesn't mean it might not be suitable in certain
cases if you know the tradeoffs.

-- David