[Twisted-Python] Conch/insults -- using HistoricRecvLine

[Jean-Paul Calderone]

On Sun, 09 Dec 2007 01:21:27 +0100, Bjoern Schliessmann <chronoss at gmx.de> wrote:
>Hello,
>
>I'm using a "standard" Twisted setup here: a ServerFactory which has
>basic.LineReceiver subclass as protocol for handling connections;
>I'm using methods like connectionMade and lineReceived.
>
>Now I tried the example on HistoricRecvLine from Twisted Conch and I
>like it. I tried my protocol to subclass HistoricRecvLine directly
>instead of basic.LineReceiver, but the server won't work anymore
>(it complains about not finding certain methods as soon as a
>connection comes in). So HistoricRecvLine doesn't seem to be
>compatible to standard Core protocols.

Indeed, it is not compatible.  Despite being an apparently similar thing,
it is actually drastically different.  It may have been a mistake to give
it methods such as "lineReceived", mirroring LineReceiver.

LineReceiver takes a stream of bytes and splits them on a delimiter,
giving each element received to the "lineReceived" callback.  It also
has a convenience method, "sendLine", which writes the given bytes to
the transport and then writes the delimiter.

This is the full extent of its behavior with regard to the network.

HistoricRecvLine does a number of things.  Actually, as you pointed out,
HistoricRecvLine is just one of the objects which must be used in order
to achieve the readline-like behavior it offers.  For the purposes of this
explanation, I'm going to lump them all together and just talk about them
as HistoricRecvLine though.

First, it interprets telnet command sequences.  This means it can do
things like know how big the client's terminal is, negotiate about
linemode behavior, and so on.  This means bytes received over the network
won't always just be split into nicely delimited strings and delivered to
application code: some of them will be consumed internally.  Reversely,
it will write out bytes which are not lines being sent to the client, but
telnet command sequences indicating some requirement or response to the
telnet client.

Next, it interprets vt102 command codes.  This happens after the telnet
command sequences have been removed.  The telnet layer is a protocol, but
it acts as a transport for the vt102 layer.  This layer allows such things
as the cursor to be repositioned, the screen to be cleared, function keys
(such as control, arrows, home, F1, etc) to be interpreted and handled
separately from "normal" input.  Again, this all results in bytes being
sent and received over the socket which have nothing to do with the "lines"
which will eventually be delivered.

Finally, the readline-like behavior is implemented as yet another protocol
on top of the vt102 layer which is acting as yet another transport.  This
means you have readline stacked on top of vt102 stacked on top of telnet
stacked on top of TCP.  Finally, at this layer, eventually some of the bytes
received over the socket will be turned into a line of input which is
delivered via the "lineReceived" method.

In addition, the readline-like layer has to be very careful about what it
sends, since the prompt interface is only preserved by careful management
of the cursor position and other terminal state (ie, what is displaying on
it).  If code simply writes text data to the transport, it will end up in
the prompt somewhere, corrupting the readline interface.

>
>I had a closer look at the demo_recvline.py example and, besides
>under ten lines of real code defining functional stuff, I saw a
>real mess of wrapper code. Several different classes are used to
>hook up the DemoRecvLine class into the ServerFactory. (This simple
>example is, IMHO, a huge difference to the elegance of simple
>Twisted Core examples.)

Yes, it's quite a mess.

>
>Those wrapper classes are documented in a way that you don't
>understand what one class does unless you understand what some few
>other classes and interfaces do. Those others are documented in a
>similar way ;) There is one tutorial for conch which is in my view
>completely unrelated.

Indeed, the tutorial doesn't go into this area.  There is currently no
prose-style documentation for these APIs.

>
>No matter what, I tried to adapt and/or wrap my
>ex-basic.LineReceiver-and-now-HistoricRecvLine subclass in a way
>that I can hook it up to my ServerFactory class, but to no avail
>for hours. That's why I'm asking here. Could you please tell me
>what exactly I need to do to "convert" my new HistoricRecvLine
>subclass in a way that ServerFactory will accept it? Or is there
>even a simpler way to get basic readline-like functionality in
>Twisted? Because that's what I'm aiming at.

There may be other ways (for example, you could run a readline-using
process as a child and allow the network client to interact with it).
Whether any of them are simpler, I'm not sure.

I haven't had a chance to work with this part of Twisted much recently,
but after my most recent attempt, I think I decided that the approach
taken by HistoricRecvLine doesn't lead to easily extensible or reusable
code.  I had been investigating an API based on an explicit buffer object
which allowed different components to be isolated from each other.  You
can take a look at the incomplete results of this (again, no documentation,
sorry) here:

http://twistedmatrix.com/trac/browser/sandbox/exarkun/invective/trunk/invective/widgets.py

LineInputWidget serves roughly the same purpose as HistoricRecvLine.  It
takes a rather different approach, though.  I haven't followed this
investigation through to its conclusion, so I can't say if this approach
ultimately works any better or not.  I'm hopeful, at least. :)

If you want to stick with HistoricRecvLine, then the only suggestion that
comes to mind to make is for you to keep your original LineReceiver sub-
class separate from your HistoricRecvLine subclass.  Give a reference to
the former to the latter: have the HistoricvRecvLine subclass's lineReceived
method call lineReceived on the LineReceiver subclass (which may not need
to be a LineReceiver subclass anymore); give the LineReceiver subclass a
transport which knows how to interact with the terminal in a way which is
not disruptive to HistoricRecvLine's use of it.

If you have any other questions, feel free to ask, particularly if they're
more specific, as those will be much easier to answer. :)

Jean-Paul