[Twisted-Python] What is the minimum effort solution to make inetd-managed twisted-based application?

Tue Jul 13 22:31:46 EDT 2010

On 13 Jul, 04:52 pm, twisted-web at udmvt.ru wrote:
>On Tue, Jul 13, 2010 at 02:37:08PM -0000, exarkun at twistedmatrix.com 
>wrote:
>>On 12:57 pm, twisted-web at udmvt.ru wrote:
>> >What is the minimal effort method for building protocol instance 
>>(maybe
>> >out of
>> >already implemented protocol factory) using a transport, that uses
>> >parent-inherited sockets (or any other already connected sockets) ?
>> >I haven't yet found any single-line solution for that.
>>
>>This isn't supported, but we'd like to support it.
>
>Oops, the Bad news.
>>  There's a ticket in
>>the issue tracker, <http://twistedmatrix.com/trac/ticket/4387>, 
>>related
>>to this (but with a somewhat wider scope).
>
>Well, that feature will benefit from the feature I need. By the way, 
>twisted
>architecture and abstractions allow one (for a limited number of 
>protocols)
>to transfer the state along with the FD of the opened connection.
>Say, one may want to read HTTP headers and then send
>the connection FD to another process along with the read headers, so it 
>will
>receive and "re-read" them with dataReceived(). Modification of HTTP 
>protocol
>code will not be required here. But we need a modified transport 
>constructor.
>> >Right now I am looking at t.i.unix.Connector and t.i.unix.Port to
>> >understand
>> >how do transports get constructed by them, but well, that is too
>> >complex for a single evening.
>> >Should I really get into the details of implementing my own transport
>> >(or their constructors)
>> >to do what I need? I'm sure there should be something, that I missed 
>>in
>> >the documentation
>> >(or in the code?).
>>
>>Despite not appearing to be private, things like t.i.unix.Connector 
>>and
>>t.i.unix.Port aren't really intended for use by applications.
>
>No, no, I'm only using them as a documentation about what transport 
>looks like
>from the inside of the twisted, to other parts of the framework.
>>
>>This is the right part of Twisted to be looking at if you want to
>>contribute a patch which adds this feature, though.  And I encourage 
>>you
>>to do that. :)
>
>Well, I was trying to get into it during last evening, but it turned 
>out to be
>more appropriate to spend a week or two on that process. It looks, like
>transport abstraction is not documented as a whole, only the user 
>visible
>parts, or better say, protocol visible parts. I am still not sure 
>whether
>it is advisable to subclass a base.BaseConnector or we can simply
>create a transport instance without that "complications". It looks like
>BaseConnector contain some portion of code, that should be reused.
>>
>>The implementation should be fairly straight-forward.  Most things 
>>like
>>Port and Client and Server have a "createInternetSocket" method.  All
>>that's really necessary to use an externally created file descriptor 
>>is
>>get "createInternetSocket" to return that descriptor instead of 
>>creating
>>a new one (or skip the call to the method entirely and just use the
>>descriptor you have already).
>>
>>The biggest question I have is what the API should look like.  Somehow
>>the file descriptor needs to get from your application code (which 
>>knows
>>that inetd put an open TCP connection on fd 0) to the
>>Port/Client/Server.
>
>That depends. Are we going to limit that feature to socket FDs only, or
>should we provide more specific interfaces for different type of files?

Ultimately it would be nice to support it for each transport type 
Twisted supports.  That doesn't mean we have to do it all at once 
though. :)
>
>Two (pipe) FDs may be used to construct full-duplex half-closeable 
>transport too.
>Or may someone be interested also in master pty based transport? That 
>would indeed require some
>terminal-specific support to be present in the transport instance, 
>wouldn't it?
>Like transport.sendBreak()
>Perhaps one may use a full-duplex transport constructed from a tty 
>/slave-pty FD?
>It may be useful to provide analogous interface for half-duplex 
>connections too, IMHO.

Actually, in twisted/internet/process.py ("Do NOT use this module 
directly"), you'll see ProcessReader and ProcessWriter which are 
transports for pipes/ptys.  The normal way to access these is via 
reactor.spawnProcess.  But you'll notice that they even accept a fileno 
as an __init__ argument, unlike most other transports in Twisted.

Mirroring this on the opposite side is twisted/internet/_posixstdio.py, 
which is a standard io transport, and also accepts filenos to its 
__init__.
>I think it may look like
>  connection = somenewmodulename.socketConnection(fd, protocolFactory, 
>addrFamily, sockType, reactor=None)
>or
>  import socket
>  connection = somenewmodulename.socketConnection(socket.fromfd(fd, 
>addrFamily, sockType), protocolFactory, reactor=None)
>
>Create a transport object and associate/build a new protocol instance 
>with/on it.
>For other FD types there should be different functions, returning 
>instances
>of other types.
>
>Would you comment on that?

Something to keep in mind here is that it may be desirable to support 
Windows at some point.  This is one reason why it may make sense to have 
these APIs as methods on the reactor: the reactor can already easily be 
replaced by alternate implementations to cater to different platforms.
>>
>>The obvious options, adding another argument to
>>listen/connectTCP/UNIX/etc, would work, but is somewhat ugly (you have
>>the issue that existing mandatory arguments would just be ignored).
>>
>>Another idea would be adding new methods entirely.  I don't know if
>>that's much better, though.
>
>Adding new methods to reactor? I thought it can live completely 
>separate
>from the reactor code and do not pollute reactor interface. Reactor is 
>only
>used in the process of attaching a FD to a transport when new transport
>registers it's FDs in the reactor, so it can be an (optional?) argument
>to a constructor. However, yes, API consistency may suffer from that.

If it's possible to do this, it's probably a good idea.  Just keep in 
mind that Windows doesn't use FDs for lots of things.  StandardIO shows 
that the API inconsistency is annoying but not fatal.  Plus, we're 
working on an API which can be a layer on top of this to make it all 
consistent again (twisted.internet.endpoints).
>>
>>So, if we can come up with a nice API, I think this will be a pretty
>>quick feature to implement.
>
>I think it could be a set of functions inside a platform-dependent 
>module
>if that functions will be FD-type dependent.
>> >And by the way, I haven't found any socketpair(2) usage in the 
>>twisted
>> >framework (except for tests),
>> >how can that be? Transport based on socketpair sockets will have the
>> >same
>> >implementation, as I need. Is it true, that nobody in twisted 
>>community
>> >uses
>> >anonymous preconnected sockets in real life?
>>
>>Apparently. :)
>
>Oops, surprise!..
>BTW creating a socketpair and forking and then communicating via PB 
>seems
>like an elegant and convenient feature to me. Perhaps non-portable?

Windows doesn't have fork, so yea, you'll run into problems there. :) 
There are also problems with the way fork interacts with kqueue and 
epoll.  The usual approach is reactor.spawnProcess to create an entirely 
new Python process, and then either connect to it over TCP or just use 
stdin/stdout to talk to it.
>> >
>> >PS: I need socket-based transport, that is, full-duplex, half-
>> >closeable, with support
>> >of getting the remote endpoint address and with ability to start TLS 
>>on
>> >top of it
>> >and without implementing every that feature myself :)
>>
>>Hopefully if we can figure out how to create a Twisted transport 
>>object
>>from an existing file descriptor, you should have no trouble with the
>>rest of these.
>
>That's what I need, and I really hoped, that it is already implemented. 
>What a surprise.
>Surely, I would not mind contibuting to such a great project, but first 
>I
>would have to get more understanding of it's abstractions.

I'll answer whatever questions I can. :)

Jean-Paul