[Twisted-Python] Problems with PB and Jelly...

Jasper Phillips jasper at peak.org
Mon Mar 24 18:02:17 EST 2003

Hash: SHA1

On Sun, 23 Mar 2003, Brian Warner wrote:

Thanks for the response!

[snip: pb.Copyable copies all references found]

> > However, I'm sometimes getting an exception when the actual dictionary
> > copying is done, as something other than a dict is being copied into
> > __dict__.  At this point the "jelType" is "dereference"...
> If it's something else, let us know (and provide a small test case??) so we
> can fix it at the sprint.

Here's the relavent end of the error:
File "..\twisted\spread\flavors.py", line 386, in setCopyableState
  self.__dict__ = state
exceptions.TypeError: __dict__ must be set to a dictionary

I've included a test case at the end of the message.

> > Damn, it looks like this might be the cuplrit.  "reference" jelyTypes are
> > recursively descended into before they are stored, and if a dereference is
> > found before it's stored... some sort of _Dereference object is created?
> > An attempt is then made to copy this into __dict__, and boom.
> Yes, the current Jelly code looks for objects that are referenced multiple
> times in the same jellying call and marks them with "reference" tags. When
> another reference to the same object is detected, it is jellied with a
> "dereference" tag that points to the earlier "reference" marker. The "cook",
> "prepare", and "preserve" methods are used to implement these multiple
> phases. Circular or recursive references are handled because the reference
> number is allocated when we start to jelly the object, even though the state
> is not yet known.

The number is allocated earlier on, but it isn't stored for use by
"derference" jelTypes until after recursion, as I will attempt to describe:

Here are the relavent parts of jelly._Unjellier.  Note that the reference is
stored after it's value is recursively computed -+
def _unjelly_reference(self, lst):               |
    refid = lst[0]                               |
    exp   = lst[1]                               |
    o     = self.unjelly(exp)           <--------+
    ref   = self.references.get(refid)  <--------+
    if (ref is None):
        self.references[refid] = o
    elif isinstance(ref, NotKnown):
        self.references[refid] = o
        assert 0, "Multiple references with same ID!"
    return o

This ends up causing the following method to return a _Dereference, rather
than a state dictionary that unjelly() would normally return.  This then
results in the error I list above.

def _unjelly_dereference(self, lst):
    refid = lst[0]
    print refid #!!! added
    x = self.references.get(refid)
    if x is not None:
        return x
    der = _Dereference(refid)
    self.references[refid] = der
    return der

> This scheme will change on Tuesday. The "reference" tags will go away and be
> replaced by an implicit marker that is notionally inserted every time we
> start jellying a new mutable object. The "dereference" tags will then point
> to these implicit markers. This should improve performance quite a bit, and
> will pave the way to a combined jelly+banana extension module that should
> give an enormous speedup (doing everything in C).

Sounds good!  No point in my suggesting a fix then. ;-)  Will a new version
of twisted be released along with these changes, or would I have to go
through CVS?


Here are 3 files for reproducing the "Unjellying a Circular Reference Bug".
Just stick them all in the same place, and start the server then the

- -Jasper

Listing for objectBug.py
from twisted.spread import pb

class ClassA( pb.Copyable, pb.RemoteCopy ):
    def __init__( self ):
        self.ref = ClassB( self )

class ClassB( pb.Copyable, pb.RemoteCopy ):
    def __init__( self, ref ):
        self.ref = ref

import sys
pb.setCopierForClassTree( sys.modules[__name__], pb.Copyable )

Listing for serverBug.py
#/usr/bin/env python
from twisted.internet import app
from twisted.spread   import pb
from twisted.cred     import authorizer
import clientBug, objectBug

class MyPer( pb.Perspective ):
    def attached( self, client, identity ):
        self.client = client
        return pb.Perspective.attached( self, client, identity )

    def perspective_receive( self, obj ):
        self.client.callRemote( "receive", obj )

class MyService( pb.Service ):
    perspectiveClass = MyPer

    def __init__( self, serviceParent, auth ):
        pb.Service.__init__( self, "MyService", serviceParent, auth )

def startServer():
    myApp = app.Application("pbServer")
    auth  = authorizer.DefaultAuthorizer( myApp )
    s     = MyService( myApp, auth )

    s.createPerspective( "player1" ).makeIdentity( "password1" )

    myApp.listenTCP( 9000, pb.BrokerFactory( pb.AuthRoot( auth )))
    myApp.run( save=0 )

if __name__ == '__main__':

Listing for clientBug.py
#/usr/bin/env python
from twisted.internet import reactor
from twisted.spread   import pb
import objectBug

class MyClient( pb.Referenceable ):
    def connect( self, ipAddress, port, user, password ):
        defer = pb.connect( ipAddress, port, user, password,
                            "MyService", client=self, timeout=3 )
        defer.addCallback( self.connected )

    def connected( self, perspective ):
        perspective.callRemote( "receive", objectBug.ClassA() )

    def remote_receive( self, obj ):
        print "Object received back"

def startClient():
    client = MyClient()
    client.connect( "localhost", 9000, "player1", "password1" )

if __name__ == '__main__':

Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org


More information about the Twisted-Python mailing list