'conch' or 'cftp' with no known_hosts file tracebacks, hangs

[robertc]

In conch/client/default.py, setting the known hosts output content lines to :
===
        known_hosts = open(os.path.expanduser('~/.ssh/known_hosts'), 'a+')
        if known_hosts.tell() > 1:
            known_hosts.seek(-1, 2)
            if known_hosts.read(1) != '\n':
                known_hosts.write('\n')
===
will fix running cftp with a missing known_hosts file.

[exarkun]

#1588 was a duplicate of this.

[glyph]

Here's an example of conch dying on this:

glyph@alastor:~/.ssh$ rm known_hosts
glyph@alastor:~/.ssh$ conch localhost
The authenticity of host '127.0.0.1' can't be established.
RSA key fingerprint is 1f:2c:e1:01:ed:77:65:fd:8b:fb:5e:b4:60:d8:dd:70.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.
Traceback (most recent call last):
  File "/home/glyph/Projects/Twisted/trunk/twisted/python/log.py", line 51, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/glyph/Projects/Twisted/trunk/twisted/python/log.py", line 36, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/glyph/Projects/Twisted/trunk/twisted/python/context.py", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/glyph/Projects/Twisted/trunk/twisted/python/context.py", line 37, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/home/glyph/Projects/Twisted/trunk/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "/home/glyph/Projects/Twisted/trunk/twisted/internet/tcp.py", line 463, in doRead
    return self.protocol.dataReceived(data)
  File "/home/glyph/Projects/Twisted/trunk/twisted/conch/ssh/transport.py", line 314, in dataReceived
    self.dispatchMessage(messageNum, packet[1:])
  File "/home/glyph/Projects/Twisted/trunk/twisted/conch/ssh/transport.py", line 329, in dispatchMessage
    f(payload)
  File "/home/glyph/Projects/Twisted/trunk/twisted/conch/ssh/transport.py", line 996, in ssh_KEX_DH_GEX_REPLY
    d = self.verifyHostKey(pubKey, fingerprint)
  File "/home/glyph/Projects/Twisted/trunk/twisted/conch/client/direct.py", line 91, in verifyHostKey
    fingerprint)
  File "/home/glyph/Projects/Twisted/trunk/twisted/conch/client/default.py", line 46, in verifyHostKey
    known_hosts = open(os.path.expanduser('~/.ssh/known_hosts'), 'r+')
exceptions.IOError: [Errno 2] No such file or directory: '/home/glyph/.ssh/known_hosts'

[glyph]

(In [25365]) Fixes for the conch client's known_hosts parsing and host key verification.

Fixes #1376 Fixes #1301 Fixes #3494 Fixes #3496 Fixes #1292 Fixes #3499

The main thrust of this change is that it adds a new module to conch, twisted.conch.client.knownhosts, which provides a structured representation of OpenSSH's known_hosts file. This is a big step in the direction of modularizing conch's support for verifying host keys and storing the results of that verification, although the internal connection APIs are mostly unchanged at this point, they could now easily be adjusted to speak to an abstract interface and still manipulate the user's actual known_hosts entries rather than completely overriding them.

The individual bugs which this change fixes are too numerous to bother describing twice; have a look at the tickets themselves more information. Suffice it to say that the conch client is now a lot more reliable (although it still has a long way to go).

In addition to making the conch client more stable, the new API allows for some programmatic manipulation of known_hosts files, which might be independently useful.

Additionally, although the review notes some ways in which the tests can still be improved, the tests for this module can be cited as a much better example for conch contributors as to how unit tests within conch should look. While these tests are covering almost exclusively new code, they are covering functionality which has existed (and been broken) for quite some time.

Author: glyph

Reviewers: exarkun, jml