[Twisted-Python] Unjelly - recursion limit reached

David K. Hess dhess at verscend.com
Mon Nov 7 07:42:34 MST 2005


On Nov 5, 2005, at 8:34 PM, Brian Warner wrote:

>> When a view_ method is called in a cacheable and the return value  
>> is a
>> graph of interconnected cacheables, is the return value completely
>> serialized at that point?
>
> Do you mean that some view_ method is being invoked on a  
> pb.Referenceable (or
> pb.ViewPoint, I suppose) ? And that method is returning an instance  
> that
> inherits from pb.Cacheable? And this pb.Cacheable instance holds  
> references
> to other pb.Cacheables? Which may or may not have been transmitted  
> across
> this wire once already?

Yes, you expertly filled in the gaps of my original question. :-)  
That's exactly what I'm asking.

> The general answer is yes, the return value is completely  
> serialized right
> then. The serialized form of the return value is transmitted in  
> pieces as
> jelly recurses down the retval object graph, each new node visited  
> causes a
> little bit more data to be sent. However, it does not stop for  
> anything, and
> there is no way for an object being serialized to indicate that it  
> wants to
> put off serialization for a while[1]. On the receiving end, the
> callRemote()'s Deferred will not be fired until the retval has  
> completely
> finished deserialization. Intermediate objects may be constructed  
> while
> deserialization is taking place, but that ought to be invisible to the
> caller.

That's a relief. And I shouldn't have expected a different answer.  
Twisted PB is excellent work and that would have been a glaring  
oversight.

> I'm not sure what would happen if, say, your  
> getStateToCacheAndObserveFor
> method did a callRemote though the same wire. newpb has a queue to  
> handle
> this sort of thing (the callRemote doesn't get transmitted until at  
> least
> after the current operation has finished), but I don't know what  
> oldpb does.
> If it isn't clever enough to be reentrant, the receiving end will get
> interleaved object state from the two operations and the results  
> will be very
> very messy.
>
> getStateToCacheAndObserveFor and getStateToCopy are called from PB  
> internals,
> and as a result it may not be safe to make other PB calls from  
> there. view_*
> is called when the wire is in a stable state (i.e. remote methods  
> calls are
> top-level objects on the wire), so I think it should be safe to make
> arbitrary PB calls from it.

All of my getStateToCacheAndObserveFor are very simple with no other  
calls. :-(

>> I'm concerned that my hierarchy of cacheables received by the client
>> is inconsistent because it is being modified by another client view_
>> call into the server before the previous hierarchy has been fully
>> serialized to the first client.
>
> Note that each callRemote is more-or-less atomic, and the creation  
> of a whole
> object graph is more-or-less atomic, but it is entirely possible  
> that other
> method calls will happen during the middle of a huge slew of  
> RemoteCache
> updates. So if object A modifies a dozen Cacheables at once, and  
> object B has
> a RemoteCache that is watching them, object C might sneak in a  
> callRemote
> while only half of the updates have been processed. The same thing  
> will
> happen if your Cacheable update method does multiple callRemotes to  
> do its
> job.

I'm not sure I'm following this fully. My situation is one server  
connected to many clients. My assumption is that remoteCall results  
and observer updates are serialized and kept in order in a qeueue to  
each client. In other words, I assume that if client A connects and  
performs a call that results in a Cacheable graph that takes a while  
to serialize and deliver to client A, even if a another client B  
calls in and modifies the state of one of the Cacheables that was  
already serialized in the result being delivered to client A, the  
update to that particular Cacheable will be properly queued behind  
the original result still being delivered to A.

Bottom line, if all of these things are true, then I'm not sure what  
is going wrong. Is it possible that jelly is at times walking my  
object graph (which is interconnected to a certain degree and changes  
dynamically) in such a way that it has to perform a significant  
amount of recursion? In other words, maybe this isn't a bug but I  
just need to increase the runtime recursion limit?

FYI, here's a snippet of the exception:

------
   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 540, in unjelly

   File "twisted\spread\flavors.pyo", line 394, in unjellyFor

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 540, in unjelly

   File "twisted\spread\flavors.pyo", line 394, in unjellyFor

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 621, in _unjelly_reference

   File "twisted\spread\jelly.pyo", line 540, in unjelly

   File "twisted\spread\flavors.pyo", line 451, in unjellyFor

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 540, in unjelly

   File "twisted\spread\flavors.pyo", line 394, in unjellyFor

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 540, in unjelly

   File "twisted\spread\flavors.pyo", line 394, in unjellyFor

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 621, in _unjelly_reference

   File "twisted\spread\jelly.pyo", line 540, in unjelly

   File "twisted\spread\flavors.pyo", line 451, in unjellyFor

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary

   File "twisted\spread\jelly.pyo", line 603, in unjellyInto

   File "twisted\spread\jelly.pyo", line 553, in unjelly

   File "twisted\spread\jelly.pyo", line 646, in _unjelly_list
----

Thanks for your insight!

Dave




More information about the Twisted-Python mailing list