[Twisted-Python] Serial interface for software program

[anatoly techtonik]

Hello, network hackers,

Recently I've got a Raspberry Pi and a friend showed me its serial
interface and helped to setup it. I was stunned. What I saw - you open
serial terminal, and you jump in into the middle of boot section. Just
open a terminal and you see what's going on inside of R.Pi. Any time.
That's was an intro. =)

For a long time I wanted the same interface for debugging software
programs. When I was hacking on Spyder I found that my human brain is
too limited to squeeze details of dynamic of interaction of objects at
run-time. Spyder is written in Python with Qt, and it is a parallel
application much like any Twisted app is. But I wanted to push the
progress, I have to deal with complexity, so I badly needed to be able
to draw a video of interaction at run-time. That was the problem I
tried to solve.

And for the first step I needed a way to connect to Spyder at run-time
and see what's going on. I didn't know how to call that way before,
but now I have a rather accurate wording - "a serial interface for an
application". That is the background for the question I am trying to
ask.

What is the best way to implement such interface considering the
following properties:
1. it needs to be simple (so that you can quickly create python script
that reads the info)
2. it needs to be universal (so that script for one app worked for other)
3. it should be asynchronous (connected terminal should not degrade performance)
4. it needs to be reliable (at least detecting missing packets)
5. and cross-platform (and pure python)

Why Twisted? For the first I think that this problem is actual for
people who are dealing with complexity of dynamic and interconnected
systems built with the help of Twisted. For the second I believe it
can only be solved or analyzed by people skilled in interprocess and
internetwork communications.

One button test:
1. program behaves weird
2. user hits the button
3. terminal pops up
4. shows a lot of data for a running program

I am interested to know.
1. how the button should discover the program
network port seems cool, but what about two parallel programs? what
about different programs that provide this interface?

2. how terminal receives the data
i see it as line based - one line - one event, everything is text

3. how to make it fast
TCP clearly won't here

4. how to send the data
what if two threads (or parallel code) produce serial data? who should
run the server that serves connected terminals, and how other thread
will send message to this server? you can not run two TCP servers on
the same port on the app side, so is there an alternative? how to
detect missing data in a stream? how to avoid dealing with incomplete
packets and low level network details?

Thanks. I'd like to get back to hacking on Spyder one day, but this is
a stumbling block for me every time I think about it or any program of
compared complexity.
-- 
anatoly t.