[Twisted-Python] Endpoint composition syntax

[Cory Benfield]

> On 7 Sep 2016, at 00:05, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> 
>> On Sep 6, 2016, at 11:55 AM, Tristan Seligmann <mithrandi at mithrandi.net <mailto:mithrandi at mithrandi.net>> wrote:
>> 
>> Currently there is no way to explicitly compose Twisted endpoints, but several endpoint implementations have arisen that explicitly wrap another endpoint, and so have needed a way to do this. So far, this has been implementing by passing in an endpoint description, and then calling serverFromString/clientFromString internally in the endpoint to construct the wrapped endpoint. I've seen two different ways of encoding the "inner" endpoint:
>> 
> 
> Some other points to keep in mind for this discussion:
> 
> We may want a syntax that supports composing multiple endpoints, not just 2.
> The existing syntax is kind of crummy; ":" as a separator has serious problems, considering its presence in both URLs and IPv6 literals.  I wouldn't say we should necessarily re-design the whole syntax to accommodate this, but just having a whole new syntax might not be a bad thing either.

I mentioned this casually to Tristan in IRC, but the current syntax and use of endpoints in Twisted gets close to a quite profound idea about protocol nesting that is lurking in the space of convention. Extending the endpoint syntax to have a blessed way of essentially composing endpoints together gives the potential of using the endpoint syntax to design and implement various “tunneling” features that are very useful.

If we take the arrow syntax, for a moment, you could conceive a truly insane client wanting to write an endpoint to run FTP over that’s a bit like:

    tcp:host=someftp.server:port=21->http:verb=connect->tcp:->socks5:targetname=mypersonalhttpproxy.server->tcp:host=mycorporatesocksproxy.server:port=2121

This would represent tunneling FTP over TCP over HTTP over TCP over SOCKS over TCP. For extra fun you can throw in some Tor.

Alternatively, and quite a bit more realistically, you could have a userspace SCTP implementation that supports being tunnelled over UDP. In this instance, rather than write a single “sctp-over-udp” endpoint, you could write a generic sctp endpoint that, if it is composed with another endpoint, expects that endpoint to provide a datagram-style transport to it.

All of this is rather pie in the sky, and potentially the purest example of YAGNI that it’s possible to imagine. However, it’s worth highlighting that endpoints are inches away from being a really powerful composable tool for saying “tunnel this protocol over this other protocol”. We may not *want* that, but it’s an interesting thought.

Cory

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20160907/b9a0f39b/attachment-0002.html>