StdioOnnaStick ignores sys.stdout.encoding, won't print unicode properly

[termie]

Twisted overloads stdout to go to its log, but the log ignores encoding and tends to choke on unicode even though my terminal can support it

[termie]

A patch to have StdioOnnaStick default to the sys.stdout.encoding and push that through to msg()

[exarkun]

Needs tests. I also don't think this is the right way to go about this. StdioOnnaStick can encode any unicode it sees: the rest of the logging system doesn't need to know anything about this.

[termie]

glyph, safe_str is at fault here, too, as it by design only wants to handle ascii. So either safe_str should handle everything as unicode (which would be unsafe to most terminals and choke japanese), it should not be used, or it needs to have its encoding be

As StdioOnnaStick has a known encoding, sys.stdout.encoding, it is safe for things going through there to use that encoding, however other logging applications do not have the benefit and as long as msg() uses safe_str, which fails on different encodings, and nothing accepts an encoding argument, nobody gets a chance to output proper data.

[exarkun]

If the purpose here is to fix `print', then my above comments apply.

If the purpose is to add unicode support to twisted.python.log, then there's a bit more work to do. StdioOnnaStick isn't the right place to determine the encoding, since the log may not be going to stdout. FileLogObserver does the actual write, so it should do the actual encoding. safe_str shouldn't know about encodings, but maybe it should return a unicode string or there should be a safe_unicode. The caller can worry about doing the encoding.

This all has the benefit of avoiding mixing byte strings with different encodings and passing an explicit encoding name through the logging system.

[termie]

A patch fixing only the StdioOnnaStick side of things

[exarkun]

The patch should include unit tests. Thanks.

[davidsarah]

The absence of an encoding attribute on StdioOnnaStick instances caused  Tahoe-LAFS bug #1099.

[itamarst]

(In [32003]) Branching to 'stdio-unicode-1990'

[itamar]

I wrote a bunch of tests for StdioOnnaStick, and then added the encoding attribute as well as code to encode unicode. The limited goal is to make print continue to work in the face of Twisted logging; if printing unicode used to work, it should continue to work.

[therve]

1. You should use addCleanup in the tests, instead of try/finally.

2. It would be nice to use better variable names than f and s.

3. You could reduce the number of calls to sys.getdefaultencoding by making the default encoding argument to StdioOnnaStick.__init__ None, and use getdefaultencoding in that case.

Build results here:  http://buildbot.twistedmatrix.com/boxes-supported?branch=/branches/stdio-unicode-1990

Thanks, please merge!

[itamarst]

(In [32022]) Merge stdio-unicode-1990

Author: termie, itamarst Reviewer: therve Fixes: #1990

When the log system overrides sys.stdout and sys.stderr, continue to support standard Python behavior of encoding Unicode in print.