[Twisted-Python] Effects of calling transport.writeSomeData() ?

[exarkun at twistedmatrix.com]

On 20 Jun, 10:22 pm, mark at catseye.org wrote:
>We're writing a Twisted 14.0.0 application (on Python 2.7.7, Mac OS 
>10.9.3) that uses Conch as an SSH client; this is working fine. 
>However, we have the requirement that in an advanced mode of operation 
>for power users that the application take advantage of OpenSSH 
>connection multiplexing over an already-established-by-the-user OpenSSH 
>ControlMaster session (via an OpenSSH ControlPath socket) instead of 
>using Conch.
>
>OpenSSH requires its new session command and forwarded file descriptors 
>to be sent over the socket in a very particular way: the command must 
>be sent first, followed by message with a '\0' byte with each forwarded 
>file descriptor.  OpenSSH ignores the '\0' for each file descriptor, 
>extracting the file descriptors themselves from the message's ancillary 
>data.
>
>The following will not work because none of the calls to write() send 
>their data until control is returned to the reactor, while 
>sendFileDescriptor() queues up the descriptors such that they get sent 
>them with the very next data that is sent -- which wind up being the 
>first three bytes of the command rather than the three '\0' bytes.
>
>class OpenSSHMuxProtocol( protocol.Protocol ):
>     # built via reactor.connectUNIX()
>     def sendCommand( self, command ):
>         # Does not work:
>         self.transport.write( command )
>         self.transport.sendFileDescriptor( sys.stdin.fileno() )
>         self.transport.write( '\0' ) # payload for the stdin file 
>descriptor
>         self.transport.sendFileDescriptor( sys.stdout.fileno() )
>         self.transport.write( '\0' ) # payload for the stdout file 
>descriptor
>         self.transport.sendFileDescriptor( sys.stdout.fileno() )
>         self.transport.write( '\0' ) # payload for the stderr file 
>descriptor
>         # ^^^ Does not work
>
>
>But this next solution /does/ work:
>
>from socket import SOL_SOCKET
>from twisted.python.sendmsg import SCM_RIGHTS, send1msg
>
>class OpenSSHMuxProtocol( protocol.Protocol ):
>     # built via reactor.connectUNIX()
>     def sendCommand( self, command ):
>         self.transport.writeSomeData( command ) # data is sent over the 
>socket immediately
>         send1msg( self.transport.socket.fileno(), "\0", 0,
>             [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdin.fileno() ) 
>) ] )
>         send1msg( self.transport.socket.fileno(), "\0", 0,
>             [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdout.fileno() 
>) ) ] )
>         send1msg( self.transport.socket.fileno(), "\0", 0,
>             [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stderr.fileno() 
>) ) ] )
>
>
>My questions are:
>
>Is it bad to bypass the reactor and send data directly/immediately this 
>way using writeSomeData() and send1msg()?  Note that sendCommand() 
>actually gets

Yes.  `writeSomeData` is not a method on any transport interface.  It is 
an implementation detail of particular transports.
>called in response to a OpenSSHMuxProtocol.dataReceived() event.  If 
>bypassing the reactor this way is bad, how bad is it and what are the 
>consequences or effects?

This use is untested.  There's no reason to expect it will continue to 
work with future Twisted releases (or, really, that it fully works now; 
since `writeSomeData` bypasses the transport's buffering layer, it seems 
like you're risking an out-of-order or partial send; probably these will 
only arise under load so you may not have observed them in your 
testing).
>Is there a better way to get a working solution?  I think I'd need some 
>way to guarantee that the write of the command was actually sent to the 
>OpenSSH server before the file descriptors are forwarded -- for 
>example, if a Deferred was used whose first callback wrote the command 
>and whose second callback forwarded the descriptors, would a call to 
>the reactor to actually sent the command be guaranteed between the two 
>callbacks?

The proper way to do this would be for OpenSSH to acknowledge the 
operation.  At this point you would know it's safe to proceed to the 
next operation.  Since you didn't mention anything about 
acknowledgements, I'm guessing there are none.

Since you're already relying on `self.transport.socket.fileno()` and 
`send1msg` (basically, bypassing the transport abstraction and just 
doing socket operations yourself) one improvement you could make would 
be just to rely on that for the whole thing.  Don't use `writeSomeData`. 
Use `socket.send(command)`.  At least this way you're only relying on 
being able to treat a transport like a UNIX socket - not on the 
particulars of the transport's buffering implementation.

A different approach you could take would be to implement this 
connection sharing feature for Conch.  I can pretty much guarantee it's 
possible to implement since an older version of Conch actually did 
implement it. :)  The implementation was removed because it was fragile, 
complicated, and poorly tested.  It would be great to re-introduce the 
functionality with a higher quality implementation.

Jean-Paul