[Twisted-Python] Using StandardIO and pipes.

Maxim Lacrima lacrima.maxim at gmail.com
Mon Aug 13 08:18:32 EDT 2012


Hi everyone!

Thanks for the help! I used an attached example and after implementing
IHalfCloseableProtocol it works! Thanks a lot!

Still, Twisted seems hard to me, probably because I don't quite understand
how to use and combine Twisted's interfaces.
With your help I can now connect my program to another process using pipe.
But, for example, I want to be able to supply data to my program not only
via stdin, but also as command line arguments. I.e. instead of
$ echo foo | ./item.py

I want to do:
$ ./item.py foo

My feeling here is that I can reuse the same protocol (EchoItemProtocol)
here, but now instead of StandardIO I need my own factory (or transport, or
whatever it is named) class that knows how to read command line args and
talk the same protocol. Sure I can read arguments using sys.argv[1:], but
what do I do next to properly pass that data to the protocol? Does my class
have to implement ITransport or IProducer etc. to talk EchoItemProtocol
similar to StandardIO?
I hope my question is clear.

Thanks in advance.

On 10 August 2012 08:18, Glyph <glyph at twistedmatrix.com> wrote:

>
> On Aug 9, 2012, at 7:39 PM, Andrew Bennetts <andrew at bemusement.org> wrote:
>
> Maxim Lacrima wrote:
> […]
>
> So my conclusions are:
> 1. When I write to stdin of my script from keyboard then
> `self.transport.write()` works as expected.
> 2. When I use pipes or redirection to my script `self.transport.write()`
> doesn't produce any output.
> 3. If I replace self.transport.write() with print statements it works.
>
>
> Buffering, perhaps?
>
> As a quick test, try adjusting your callback to loseConnection on the
> transport
> after to the write, that should flush the buffers.  If buffering turns out
> to
> be your problem, and you really do expect your pipes to be at most
> line-buffered, you'll need to arrange for the buffering on the stdin and
> stdout
> file descriptors to be changed.  I don't think Twisted has that builtin,
> so you
> may need to do that before creating the StandardIO object using the regular
> Python APIs.
>
>
> It's not this kind of buffering problem.  I was actually able to reproduce
> it by replacing "Item.get()" with a deferLater.  (In the future, please
> perform these types of replacements yourself before posting your examples;
> it's much easier to help out with complete, runnable programs.)
>
> The issue is that twisted.internet.stdio considers your protocol to only
> know about one kind of connection loss: i.e. that your entire transport has
> gone away.  So, when the input is closed, the output file descriptor is
> immediately closed as well.
>
> Unfortunately - and I think this is a bug in Twisted - the input
> connection being lost causes the output connection to be immediately and
> somewhat aggressively closed.  The difference between "echo" and typing
> interactively is that you send "test_item_id\n", but echo sends
> "test_item_id\n" *EOF*.
>
> Luckily you can work around this, by explicitly asking Twisted for
> notifications about half of the connection being closed.  You do this by
> implementing IHalfCloseableProtocol.
>
> Although Twisted should be better about how it handles shutting down
> stdout, this nuance *does* reveal some logic that your code is missing:
> once your input is done, you don't have any concept of waiting for your *output
> *to be done before shutting everything down.  So, if you add both the
> IHalfCloseableProtocol implements declaration to receive the relevant
> notifications, and some logic to keep track of when all the output has been
> written, you can get exactly the behavior that I assume you want.
>
> I've attached an example that does all of these things with a simple
> timer, since this is something that Twisted probably needs to document a
> bit better.
>
>
>
> Cheers,
>
> -glyph
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>


-- 
with regards,
Maxim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20120813/2beda1d2/attachment.htm 


More information about the Twisted-Python mailing list