[Twisted-Python] What is the minimum effort solution to make inetd-managed twisted-based application?

[Alexey]

On Tue, Jul 13, 2010 at 02:37:08PM -0000, exarkun at twistedmatrix.com wrote:
> On 12:57 pm, twisted-web at udmvt.ru wrote:
> >What is the minimal effort method for building protocol instance (maybe 
> >out of
> >already implemented protocol factory) using a transport, that uses
> >parent-inherited sockets (or any other already connected sockets) ?
> >I haven't yet found any single-line solution for that.
> 
> This isn't supported, but we'd like to support it.

Oops, the Bad news.

>  There's a ticket in 
> the issue tracker, <http://twistedmatrix.com/trac/ticket/4387>, related 
> to this (but with a somewhat wider scope).

Well, that feature will benefit from the feature I need. By the way, twisted
architecture and abstractions allow one (for a limited number of protocols)
to transfer the state along with the FD of the opened connection.
Say, one may want to read HTTP headers and then send
the connection FD to another process along with the read headers, so it will
receive and "re-read" them with dataReceived(). Modification of HTTP protocol
code will not be required here. But we need a modified transport constructor.

> >Right now I am looking at t.i.unix.Connector and t.i.unix.Port to 
> >understand
> >how do transports get constructed by them, but well, that is too 
> >complex for a single evening.
> >Should I really get into the details of implementing my own transport 
> >(or their constructors)
> >to do what I need? I'm sure there should be something, that I missed in 
> >the documentation
> >(or in the code?).
> 
> Despite not appearing to be private, things like t.i.unix.Connector and 
> t.i.unix.Port aren't really intended for use by applications.

No, no, I'm only using them as a documentation about what transport looks like
from the inside of the twisted, to other parts of the framework.

> 
> This is the right part of Twisted to be looking at if you want to 
> contribute a patch which adds this feature, though.  And I encourage you 
> to do that. :)

Well, I was trying to get into it during last evening, but it turned out to be
more appropriate to spend a week or two on that process. It looks, like
transport abstraction is not documented as a whole, only the user visible
parts, or better say, protocol visible parts. I am still not sure whether
it is advisable to subclass a base.BaseConnector or we can simply
create a transport instance without that "complications". It looks like
BaseConnector contain some portion of code, that should be reused.

> 
> The implementation should be fairly straight-forward.  Most things like 
> Port and Client and Server have a "createInternetSocket" method.  All 
> that's really necessary to use an externally created file descriptor is 
> get "createInternetSocket" to return that descriptor instead of creating 
> a new one (or skip the call to the method entirely and just use the 
> descriptor you have already).
> 
> The biggest question I have is what the API should look like.  Somehow 
> the file descriptor needs to get from your application code (which knows 
> that inetd put an open TCP connection on fd 0) to the 
> Port/Client/Server.

That depends. Are we going to limit that feature to socket FDs only, or
should we provide more specific interfaces for different type of files?
Two (pipe) FDs may be used to construct full-duplex half-closeable transport too.
Or may someone be interested also in master pty based transport? That would indeed require some
terminal-specific support to be present in the transport instance, wouldn't it?
Like transport.sendBreak()
Perhaps one may use a full-duplex transport constructed from a tty/slave-pty FD?
It may be useful to provide analogous interface for half-duplex connections too, IMHO.

I think it may look like
  connection = somenewmodulename.socketConnection(fd, protocolFactory, addrFamily, sockType, reactor=None)
or
  import socket
  connection = somenewmodulename.socketConnection(socket.fromfd(fd, addrFamily, sockType), protocolFactory, reactor=None)

Create a transport object and associate/build a new protocol instance with/on it.
For other FD types there should be different functions, returning instances
of other types.

Would you comment on that?

> 
> The obvious options, adding another argument to 
> listen/connectTCP/UNIX/etc, would work, but is somewhat ugly (you have 
> the issue that existing mandatory arguments would just be ignored).
> 
> Another idea would be adding new methods entirely.  I don't know if 
> that's much better, though.

Adding new methods to reactor? I thought it can live completely separate
from the reactor code and do not pollute reactor interface. Reactor is only
used in the process of attaching a FD to a transport when new transport
registers it's FDs in the reactor, so it can be an (optional?) argument
to a constructor. However, yes, API consistency may suffer from that.

> 
> So, if we can come up with a nice API, I think this will be a pretty 
> quick feature to implement.

I think it could be a set of functions inside a platform-dependent module
if that functions will be FD-type dependent.

> >And by the way, I haven't found any socketpair(2) usage in the twisted 
> >framework (except for tests),
> >how can that be? Transport based on socketpair sockets will have the 
> >same
> >implementation, as I need. Is it true, that nobody in twisted community 
> >uses
> >anonymous preconnected sockets in real life?
> 
> Apparently. :)

Oops, surprise!..
BTW creating a socketpair and forking and then communicating via PB seems
like an elegant and convenient feature to me. Perhaps non-portable?

> >
> >PS: I need socket-based transport, that is, full-duplex, half- 
> >closeable, with support
> >of getting the remote endpoint address and with ability to start TLS on 
> >top of it
> >and without implementing every that feature myself :)
> 
> Hopefully if we can figure out how to create a Twisted transport object 
> from an existing file descriptor, you should have no trouble with the 
> rest of these.

That's what I need, and I really hoped, that it is already implemented. What a surprise.
Surely, I would not mind contibuting to such a great project, but first I
would have to get more understanding of it's abstractions.

Alexey.

-- 
с уважением,
Алексей Шпагин
системный администратор
цеха передачи данных
технического центра телекоммуникаций
ОАО "ВолгаТелеком" 
филиал в Удмуртской республике.