[Twisted-Python] verilog simulation calling out to twisted

[Bryan Murdock]

On Thu, Sep 21, 2017 at 4:07 AM, ex vito <ex.vitorino at gmail.com> wrote:
> On 2017-09-20, at 22:33, Bryan Murdock <bmurdock at gmail.com> wrote:
>
> Welcome to the club! :)

Thanks!

>> Now I have a crazy idea.  I'd like that Python code that my Verilog
>> calls out to to use twisted.  OK, actually I've already done that, my
>> Verilog calls out to my that TCP server that I wrote.  I had to spawn
>> a thread and run the server in that thread so that the Verilog could
>> continue to do stuff in parallel with the server.  It's working great.
>
> Can you clarify what "your Verilog" is? Python code running a Verilog simulation, maybe? Something else?
<snip/>
> I do not fully understand this: if "Verilog calls out to TCP server" means it establishes a TCP connection (does it?) what does "Verilog calls out to a TCP client" mean? Does it call a function/method with async Twisted code? How does it handle the fact that the result is async?

Sorry, I was worried that this would be too hard to explain and/or I
may be giving too many details and confusing the issue.  Let me try
again.  It's probably simplest to think of the whole simulation as a
TCP proxy.  It looks like this:

some TCP client process <--> Python TCP Server <--> Verilog <-->
Python TCP Client <--> some TCP server

The Verilog, Python TCP server, and Python TCP client are all running
in the same process (some TCP client and some TCP server can be a web
browser and web server or whatever).  The Verilog code starts up,
calls a python function to start the Python TCP Server as a thread,
calls another python function to start the Python TCP Client as a
thread, and then essentially just loops asking (polling) the Python
TCP Server if it has any data, if so, it calls a function to send that
data to the Python TCP Client.  It does the same in reverse to flow
data from client to server.

I originally wrote the Python TCP Client and Server as low-level
socket code, handling only one connection at a time and not
automatically reconnecting if a connection was lost.  All the
downsides of using multiple processes instead of threads that you
listed are exactly the reasons I used threads.  Verilog has no support
for spawning subprocesses or doing interprocess communication.  It
can't spawn threads either, to be fair though[1].  I have to do "real
programming" things like that by calling python functions.  Threads
just seemed easier at the time because I can just use a Queue to get
data to/from the server and client threads from/to the Verilog.  And
it all worked great.

Then, when I went to add support for multiple connections and
auto-reconnects I discovered twisted.  I rewrote the Python TCP server
and client with twisted but now data doesn't flow.  I assume that's
because the client and server are making thread-unsafe calls to the
reactor.  Is that true?

(By the way, I can run the client and server standalone as separate
processes, without verilog involved, and they work fine that way.  I
have also replaced the twisted Python TCP Client with my original
socket-based TCP client and everything works fine.  It's just when I
try to use twisted for both client and server that it fails).

> In general you are able to run concurrent (not parallel) code for TCP clients and servers within the same thread - after all, that's the key idea behind async programming with Twisted, with the reactor driving concurrent handling of multiple connections and events.

Right, I think the solution is going to have to be: have verilog spawn
one thread that runs the Python TCP Server and Client.  Or start them
both as subprocesses and figure out how to communicate with them, I
guess.

Bryan

1. Aside from it's "green threads" that it uses to model concurrent
digital logic