[Twisted-Python] Unjelly - recursion limit reached
David K. Hess
dhess at verscend.com
Mon Nov 7 09:42:34 EST 2005
On Nov 5, 2005, at 8:34 PM, Brian Warner wrote:
>> When a view_ method is called in a cacheable and the return value
>> is a
>> graph of interconnected cacheables, is the return value completely
>> serialized at that point?
>
> Do you mean that some view_ method is being invoked on a
> pb.Referenceable (or
> pb.ViewPoint, I suppose) ? And that method is returning an instance
> that
> inherits from pb.Cacheable? And this pb.Cacheable instance holds
> references
> to other pb.Cacheables? Which may or may not have been transmitted
> across
> this wire once already?
Yes, you expertly filled in the gaps of my original question. :-)
That's exactly what I'm asking.
> The general answer is yes, the return value is completely
> serialized right
> then. The serialized form of the return value is transmitted in
> pieces as
> jelly recurses down the retval object graph, each new node visited
> causes a
> little bit more data to be sent. However, it does not stop for
> anything, and
> there is no way for an object being serialized to indicate that it
> wants to
> put off serialization for a while[1]. On the receiving end, the
> callRemote()'s Deferred will not be fired until the retval has
> completely
> finished deserialization. Intermediate objects may be constructed
> while
> deserialization is taking place, but that ought to be invisible to the
> caller.
That's a relief. And I shouldn't have expected a different answer.
Twisted PB is excellent work and that would have been a glaring
oversight.
> I'm not sure what would happen if, say, your
> getStateToCacheAndObserveFor
> method did a callRemote though the same wire. newpb has a queue to
> handle
> this sort of thing (the callRemote doesn't get transmitted until at
> least
> after the current operation has finished), but I don't know what
> oldpb does.
> If it isn't clever enough to be reentrant, the receiving end will get
> interleaved object state from the two operations and the results
> will be very
> very messy.
>
> getStateToCacheAndObserveFor and getStateToCopy are called from PB
> internals,
> and as a result it may not be safe to make other PB calls from
> there. view_*
> is called when the wire is in a stable state (i.e. remote methods
> calls are
> top-level objects on the wire), so I think it should be safe to make
> arbitrary PB calls from it.
All of my getStateToCacheAndObserveFor are very simple with no other
calls. :-(
>> I'm concerned that my hierarchy of cacheables received by the client
>> is inconsistent because it is being modified by another client view_
>> call into the server before the previous hierarchy has been fully
>> serialized to the first client.
>
> Note that each callRemote is more-or-less atomic, and the creation
> of a whole
> object graph is more-or-less atomic, but it is entirely possible
> that other
> method calls will happen during the middle of a huge slew of
> RemoteCache
> updates. So if object A modifies a dozen Cacheables at once, and
> object B has
> a RemoteCache that is watching them, object C might sneak in a
> callRemote
> while only half of the updates have been processed. The same thing
> will
> happen if your Cacheable update method does multiple callRemotes to
> do its
> job.
I'm not sure I'm following this fully. My situation is one server
connected to many clients. My assumption is that remoteCall results
and observer updates are serialized and kept in order in a qeueue to
each client. In other words, I assume that if client A connects and
performs a call that results in a Cacheable graph that takes a while
to serialize and deliver to client A, even if a another client B
calls in and modifies the state of one of the Cacheables that was
already serialized in the result being delivered to client A, the
update to that particular Cacheable will be properly queued behind
the original result still being delivered to A.
Bottom line, if all of these things are true, then I'm not sure what
is going wrong. Is it possible that jelly is at times walking my
object graph (which is interconnected to a certain degree and changes
dynamically) in such a way that it has to perform a significant
amount of recursion? In other words, maybe this isn't a bug but I
just need to increase the runtime recursion limit?
FYI, here's a snippet of the exception:
------
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 540, in unjelly
File "twisted\spread\flavors.pyo", line 394, in unjellyFor
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 540, in unjelly
File "twisted\spread\flavors.pyo", line 394, in unjellyFor
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 621, in _unjelly_reference
File "twisted\spread\jelly.pyo", line 540, in unjelly
File "twisted\spread\flavors.pyo", line 451, in unjellyFor
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 540, in unjelly
File "twisted\spread\flavors.pyo", line 394, in unjellyFor
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 540, in unjelly
File "twisted\spread\flavors.pyo", line 394, in unjellyFor
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 621, in _unjelly_reference
File "twisted\spread\jelly.pyo", line 540, in unjelly
File "twisted\spread\flavors.pyo", line 451, in unjellyFor
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 654, in _unjelly_dictionary
File "twisted\spread\jelly.pyo", line 603, in unjellyInto
File "twisted\spread\jelly.pyo", line 553, in unjelly
File "twisted\spread\jelly.pyo", line 646, in _unjelly_list
----
Thanks for your insight!
Dave
More information about the Twisted-Python
mailing list