[Twisted-Python] Sending other things than strings in UDP packets

[Paul Campbell]

> Message: 5
> Date: Thu, 7 Oct 2004 08:53:08 +1100
> From: Christopher Armstrong <radeex at gmail.com>
> Subject: Re: [Twisted-Python] Sending other things than strings in UDP
> 	packets
> To: Twisted general discussion <twisted-python at twistedmatrix.com>
> Message-ID: <60ed19d404100614531935658a at mail.gmail.com>
> Content-Type: text/plain; charset=US-ASCII
> 
Christopher Armstrong <radeex at gmail.com> wrote:
> 
> Ergh. Please don't do anything _close_ to suggesting this. This is not
> the "python way", it is the "stupid, insecure,
> let-people-rm-rf-your-home-directory way".

You apparently read part of the message and then you failed to read the
next paragraph, right? At the risk of being redundant, let me reiterate:

"Read the documentation on the pickle module for more information. And be
forewarned: pickle will dump/load ANYTHING. For safety reasons, there's also
a 'safe_pickle' variant floating around."

I guess writing instructive and marginal (and probably nonworking) code for
helping out newbies is not what this mailing list is all about. I thought I
said "nonworking", "no error checking", and "lots of issues" enough times
to get the point across. Sorry, I'll "mail.compose.elitist_mode=true" next
time and give some flippant answer like "UDP sucks dude. Just use PB under
TCP."

At the risk of leaving you hanging with regards to whether such variants
actually exist, below are pointers to two that can be used off the shelf, and
possibly a third already contained within the twisted code base.

Here's one variant that includes the a "safe pickle" call:
http://zgp.org/pipermail/p2p-hackers/2004-August/002046.html

Incidentally, the protocol contains a lot of the extras that I mentioned
my stripped-down code was lacking (as well as a few small bugs). It also
handles long messages and retries as well within UDP. It has an interesting
"microprotocol" sort of structure (where each layer of the protocol builds
on the previous one). Read it in addition to the first couple functions
that handle pickling/unpickling.

Another "safe pickle" module is buried in the code for "thecircle"
(www.thecircle.org.au). Just download it and rip out the "safe_pickle.py"
module from circlelib. It is stand-alone, and designed for UDP transportation
(although not currently using Twisted).

I haven't dug really deep, but banana (part of PB) appears to be essentially
yet another incarnation of exactly the same idea. The code pattern looked
identical to the two pieces of code I just mentioned. In fact, I haven't
looked at it but I suspect that even pickle itself has the same pattern,
other than being more generalized (it will handle executables and instances,
while the safe variants will reject that).

The code for all of these modules has an identical structure. It takes
a structure and walks down it. It reads each piece and codes it in a
"Type+data" format. It rejects anything that it can't inherently decode
without aid (such as class instances). In those cases, at least the banana
variant does allow the possibility to kick it up to a higher level (via
Jelly) to handle user-level structures.

The "unpickle" code does exactly the same thing except in reverse, converting
the coded data back into a structure. There are standards (S-expressions or
XML) for the format itself but I haven't seen any truly compelling reasons to
follow those. They seem to add lots of overhead without any additional benefit.