[Twisted-Python] Using StandardIO and pipes.

Glyph glyph at twistedmatrix.com
Fri Aug 10 01:18:13 EDT 2012

On Aug 9, 2012, at 7:39 PM, Andrew Bennetts <andrew at bemusement.org> wrote:

> Maxim Lacrima wrote:
> […]
>> So my conclusions are:
>> 1. When I write to stdin of my script from keyboard then
>> `self.transport.write()` works as expected.
>> 2. When I use pipes or redirection to my script `self.transport.write()`
>> doesn't produce any output.
>> 3. If I replace self.transport.write() with print statements it works.
> Buffering, perhaps?
> As a quick test, try adjusting your callback to loseConnection on the transport
> after to the write, that should flush the buffers.  If buffering turns out to
> be your problem, and you really do expect your pipes to be at most
> line-buffered, you'll need to arrange for the buffering on the stdin and stdout
> file descriptors to be changed.  I don't think Twisted has that builtin, so you
> may need to do that before creating the StandardIO object using the regular
> Python APIs.

It's not this kind of buffering problem.  I was actually able to reproduce it by replacing "Item.get()" with a deferLater.  (In the future, please perform these types of replacements yourself before posting your examples; it's much easier to help out with complete, runnable programs.)

The issue is that twisted.internet.stdio considers your protocol to only know about one kind of connection loss: i.e. that your entire transport has gone away.  So, when the input is closed, the output file descriptor is immediately closed as well.

Unfortunately - and I think this is a bug in Twisted - the input connection being lost causes the output connection to be immediately and somewhat aggressively closed.  The difference between "echo" and typing interactively is that you send "test_item_id\n", but echo sends "test_item_id\n" *EOF*.

Luckily you can work around this, by explicitly asking Twisted for notifications about half of the connection being closed.  You do this by implementing IHalfCloseableProtocol.

Although Twisted should be better about how it handles shutting down stdout, this nuance does reveal some logic that your code is missing: once your input is done, you don't have any concept of waiting for your output to be done before shutting everything down.  So, if you add both the IHalfCloseableProtocol implements declaration to receive the relevant notifications, and some logic to keep track of when all the output has been written, you can get exactly the behavior that I assume you want.

I've attached an example that does all of these things with a simple timer, since this is something that Twisted probably needs to document a bit better.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20120809/72a869f3/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: check.py
Type: text/x-python-script
Size: 1196 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20120809/72a869f3/attachment.bin 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20120809/72a869f3/attachment-0001.htm 

More information about the Twisted-Python mailing list