_newclient does not support bare LF newlines, unlike web browsers

[ivank]

Some servers send broken HTTP responses with only \n to delimit status lines and headers, instead of \r\n. All modern web browsers support this, but _newclient does not.

Originally discovered at http://twistedmatrix.com/trac/ticket/886#comment:23

[ivank]

The only server that was found to do this was  http://news.ycombinator.com/ (as of 2009-02). Nothing else has been found so far.

[mithrandi]

 http://www.thinkgeek.com/ also has this defect, as of this posting.

[ivank]

RFC 2616 section 19.3 Tolerant Applications:

The line terminator for message-header fields is the sequence CRLF. However, we recommend that applications, when parsing such headers, recognize a single LF as a line terminator and ignore the leading CR.

[ivank]

(via  http://news.ycombinator.com/item?id=705095 , which lists other HTTP clients that had to be modified to work with LF)

[ivank]

Related bug for the old HTTP client: #2842

[jknight]

 http://www.thinkgeek.com/ no longer appears to have this problem.

But a new bad site was discovered:  http://hdl.loc.gov/loc.uscongress/legislation.111s1448

(And I can't believe ycombinator is still so broken. Sigh.)

[exarkun]

The patch on #886 appears to be a more or less complete attempt at fixing this issue. Please review.

[jonathanj]

(In [30570]) Branching to 'newclient-bare-lf-3833'

[jonathanj]

(In [30571]) Apply ivank's newclient patch for handling bare linefeeds in HTTP headers and status lines, refs #3833.

[jonathanj]

(In [30573]) Merge newclient-bare-lf-3833: Handle HTTP headers delimited by bare LFs in Agent.

Author: ivank Reviewer: jonathanj Fixes: #3833

Some non-compliant web servers send HTTP responses delimited only by \n instead of \r\n. All modern web browsers support this and now Agent does too.