[Twisted-Python] Problems with PB and Jelly...

Brian Warner warner at lothar.com
Sun Mar 23 17:22:53 EST 2003

> Hmmmm.  You're implying that the state can't hold references to other
> objects?  That might explain an exception I'm getting, which I'll describe
> below.  If this is true, is there some easy way around this, or do I need
> a custom setCopyableState()?

The state *can* hold references to other objects (as long as those objects
are themselves somehow transferrable, like pb.Copyable). I draw a distinction
between having or not having references to other objects because data that is
standalone makes more sense when transferred to other memory spaces.

Imagine your data structures make up a big directed graph. The edges are
references, each contained in one object and pointing at another. When you
copy one of the objects out to another memory space, you're plucking a node
out of the graph and putting it somewhere else. What happens to the edges?

It depends upon what flavor the referenced objects inherit from. If they are
pb.Referenceable, the edges turn into pb.RemoteReferences, and it's as if the
edge-arrows are stretched to run from the object's new location back to the
home memory space. If they are pb.Copyable, the reference is followed and the
target object copied just like the original object was. If they are neither,
you get an InsecureJelly exception. The idea is to prevent you from
accidentially copying out objects that you didn't intend to be shared.

So if the object points to a lot of other objects, those referents make up an
environment. If the environment doesn't come with the object, then that
object could be said to have a "home", and then it makes sense to talk about
the "home" version of an object versus a copy that lives somewhere "away"
from that home. If the object is mostly standalone, then it doesn't matter
where the object lives and the home/away distinction is moot.

> However, I'm sometimes getting an exception when the actual dictionary
> copying is done, as something other than a dict is being copied into
> __dict__.  At this point the "jelType" is "dereference"...

Which exception is being raised? If it's the InsecureJelly, then you're
referencing an object that doesn't inherit from one of the PB flavors. You
either need to remove that reference in your getStateToCopy() method (cut the
edge-arrow) or you need to make the referenced objects inherit from something
like pb.Copyable.

If it's something else, let us know (and provide a small test case??) so we can
fix it at the sprint.

> Damn, it looks like this might be the cuplrit.  "reference" jelyTypes are
> recursively descended into before they are stored, and if a dereference is
> found before it's stored... some sort of _Dereference object is created?
> An attempt is then made to copy this into __dict__, and boom.

Yes, the current Jelly code looks for objects that are referenced multiple
times in the same jellying call and marks them with "reference" tags. When
another reference to the same object is detected, it is jellied with a
"dereference" tag that points to the earlier "reference" marker. The "cook",
"prepare", and "preserve" methods are used to implement these multiple
phases. Circular or recursive references are handled because the reference
number is allocated when we start to jelly the object, even though the state
is not yet known.

This scheme will change on Tuesday. The "reference" tags will go away and be
replaced by an implicit marker that is notionally inserted every time we
start jellying a new mutable object. The "dereference" tags will then point
to these implicit markers. This should improve performance quite a bit, and
will pave the way to a combined jelly+banana extension module that should
give an enormous speedup (doing everything in C).

> I'll have to look into this more closely latter.  At first glance it appears
> to be something best fixed in twisted itself rather than a local
> setCopyableState()... at least to this twisted newbie. ;-)

Definitely. Twisted should "just handle" arbitrary reference graphs with no
problems right now.. the change planned for the PyCon sprint will make it
handle them faster and with less on-wire traffic than before.

> I've since looked at this code more closely as well.  The "else" branch in
> question is most definitely _not_ dead code, but it is confusing.  ;-)

I think _unjelly_instance might be dead code, because I don't see anywhere an
"instance" tag could be inserted into the stream. That might be compatibility
with an older version of the jelly side, though. Another item on the PB
sprint will be to implement proper version markers so this sort of thing can
be done properly next time.


More information about the Twisted-Python mailing list