[Twisted-Python] How to achieve a reliable "connection" over UDP?

[glyph at divmod.com]

On 09:23 am, p.mayers at imperial.ac.uk wrote:
>
>>I'm thinking basically all p2p applications will need this sort of
>>"reliability" over UDP for e.g. file transfer, so I'm hoping there is
>>something already written.
>
>http://divmod.org/trac/wiki/DivmodVertex
>
>Haven't used it, so can't comment further onit.

As it happens, I have, a little, so I can :).

To make a long story short, if you need the features that TCP provides, 
but you want the simplicity (and widespread de-jure "standard" 
implementation in routers) of hole-punching via UDP rather than complex 
dual-connection TCP startup, the easiest thing you can do is just go 
ahead and implement TCP on top of UDP.

We did this for vertex, in the "ptcp" module (pseudo-TCP) which is a 
very literal-minded implementation of the TCP specification on top of 
UDP.

It's a fully-functional proof of concept at this point, but it needs a 
lot of maintenance.  The performance is terrible because we didn't 
implement window sizing yet (the window has a static size), there are a 
few edge-cases on connection tear-down, and the tests aren't quite 
granular enough (for example, rather than testing specific latency 
cases, they just probabilistically test packet loss).  It also used a 
predecessor to the now-standard-in-twisted "AMP" protocol for 
handshaking, and should be upgraded to use the new one.  Plus, my own 
personal bugaboo: it doesn't integrate with cred properly.

However, if you need connection-oriented peer-to-peer traffic, 
contributing to Vertex is going to get you where you need to be faster 
than any ad-hoc solution.  We *have* already implemented the full TCP 
state machine and connection-management logic in Python.  I've 
personally tested that it can transfer an entire 700MB ISO image across 
the real, public internet correctly, even in its current state.

You will find willing assistance from me and probably some other Divmod 
folks & fans, because this is a project that many of us would like to 
see through to completion; we just don't have a lot of spare cycles (or 
business motivation) at the moment.

Depending on what you need, you might want to focus your maintenance 
efforts on improving PTCP's tests and documentation so as to make it an 
independently usable unit from Vertex's connection setup.  Obviously 
we'd like it if you could help us to get the whole thing to work, but I 
have to assume that by the time you're asking this question, you've 
already done some work on an application protocol to set up the 
connection.