[Twisted-Python] persistence interface

Glyph Lefkowitz glyph at twistedmatrix.com
Sun Sep 15 19:23:43 EDT 2002

Let me give some exposition of the ideas that lead to this, since the
persistence-adaptor style that Sean is proposing was at my suggestion, inspired
by what he's done so far on the Row API.

On Sun, 15 Sep 2002 14:15:29 -0400, Itamar Shtull-Trauring <itamar at itamarst.org> wrote:
> So now we have the following persistence APIs:

> 1. getStateToPersist

The idea here is that it's specifically for relational, or "essential"
persistence, not complete persistence.  In order for this kind of persistence
to work, your objects need to have a fixed list of fixed and some semantic
meaning associated with any object references: containership, ownership, or
weak reference.

So you can't return a dictionary of arbitrary data; you return a dictionary of
the attributes that the persistence layer expects to see.

This also doesn't handle object versioning, since it expects your persistence
layer to impose some sort of structure on your data.  (You can't upgrade a
relational database without some serious effort and forethought in any event.)

> 2. getStateToCopy

This is for remote mirroring.  Now, I am tempted to suggest this be used for
persistence adapters as well, since the full name of this method is
getStateToCopy*For*, and it takes the serializer object (which could quite well
be a database reflector).

However, I think the differences are more subtle than that, since as I
mentioned, information about containership will probably need to be part of the
persistence-adapter API.

Not only is it meant to have semntic parity with relational databases (at least
for its first incarnation), it's only supposed to be persisting state that an
object absolutely "needs"; any duplicated data such as cached references,
backreferences to "parent" objects, or, in the relational case, lists that
point to child objects, need not be saved.  This means that rather than the
traditional way of handling class updating in Twisted, input one state, output
a new, fixed state for new versions of the class, is unworkable.  You can
change the internal representation of the object as much as you want, as long
as you can still re-build it from the info you got from the database.

We will be able to avoid the 'for' argument in getStateToPersist because we
will have different, specialized adapter interfaces for different kinds of
persistence.  FramerD needs very different information about an object's state
than a database would, but it's likely that none of the existing state methods
will suffice.

> 3. __getstate__

This is for full, faithful replication of objects to trusted sources, such as
.tap files.  There should be as little perturbation as possible in an object's
__getstate__ method; really, the only thing this should do it provide
versioning and remove attributes which are simply not persistable.

Consider using __getstate__ for a schema migration tool: dumping a database to
a large pickle with one version of a codebase, then loading it with a new
version and undumping it back into another database with a new schema.

> 4. Whatever it is marmalade is using


This one is the one I'd think of eliminating, if we're going to think of
eliminating one.  It's rather overcomplex to produce a sensible-looking object
hierarchy with DOM, and very hard to get it right.  That's why I eventually
gave up on sensibility and just settled for correctness in Application ;).

> I don't see any difference between (1) and (3), since they both return
> dictionary that is used in persistence. Just make the requirements for
> __getstate__ to be more strict when implementing IPersistence.

That would be rather difficult, since IPersistence may want to expect things
like indexing in order to reverse properly.

> How many persistence APIs do we need?!

As many as there are persistence mechanisms that people want to use with
Twisted :-).

 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20020915/f7713295/attachment.pgp 

More information about the Twisted-Python mailing list