[Twisted-Python] Unjelly - recursion limit reached

Brian Warner warner at lothar.com
Mon Nov 21 02:02:54 EST 2005


>>> This is a pretty scary limitation. Is anybody working on a non-
>>> recursion based version of jelly? Am I going to have to move to
>>> Stackless Python?

Sorry to respond to this so late.. my Twisted time has been awfully bursty
this month.

FWIW, newpb does not suffer from this sort of stack-frame limitation. Rather
than using the C/Python runtime stack to keep track of its current place in
the object graph, it maintains a "Slicer Stack" (and corresponding Unslicer
Stack on the receiving end), which is really just a list of Slicer/Unslicer
instances. Each edge of the object graph path (from the top-most
one-per-connection Root object down to the thing currently being serialized
or unserialized) gets a Slicer/Unslicer on this list. Ridiculously deep
object graphs (like the linked-list structure you describe) will consume heap
memory for this list, but not stack frames.

The maximum number of stack frames consumed is a constant, not affected by
the shape or complexity of the data being serialized. (I haven't measured it
but I'd guess it's less than a dozen frames). The runtime memory footprint is
roughly linear with the depth of the object graph. Each Slicer contains a
generator.. I don't know how much memory is consumed to maintain the state of
a typical Slicer, but I'm not sure I'd want a million of them.

> My object graph is a natural and clean data structure. I don't want  
> to lose that just because of limitations in my tools.

PB is all about transferring your existing object graph cleanly and
accurately. newpb just does it better than oldpb. Some of the limitations of
oldpb (well, really just that 640k string limit) are there to minimize the
damage if something goes really wrong (like you connect an HTTP client to the
PB socket, and it tries to interpret "GET HTTP/1.1 index.html" as a
bajillion-byte STRING token). newpb provides better ways to specify these
restrictions (the whole 'schema' approach), so there ought to be fewer
non-user-specified arbitrary limits.

That said, some graphs are more efficient to transfer than others. A linked
list of small integers will take about three times as many bytes to serialize
as a flat list, and will consume a far greater amount of memory while doing
so. The newpb Slicers are not clever enough to do tail-recursion.

> Is this currently being addressed or in a road map? Am I unknowingly  
> volunteering for it? :-o

Well, it sounds like you might be a great candidate for beta-testing newpb
:). Keep tuned to this channel for an alpha release announcement sometime
soon.

cheers,
 -Brian




More information about the Twisted-Python mailing list