Opened 5 years ago

Closed 4 years ago

#3833 enhancement closed fixed (fixed)

_newclient does not support bare LF newlines, unlike web browsers

Reported by: ivank Owned by:
Priority: low Milestone:
Component: web Keywords: review
Cc: Branch: branches/newclient-bare-lf-3833
(diff, github, buildbot, log)
Author: jonathanj Launchpad Bug:

Description

Some servers send broken HTTP responses with only \n to delimit status lines and headers, instead of \r\n. All modern web browsers support this, but _newclient does not.

Originally discovered at http://twistedmatrix.com/trac/ticket/886#comment:23

Change History (11)

comment:1 Changed 5 years ago by ivank

The only server that was found to do this was http://news.ycombinator.com/ (as of 2009-02). Nothing else has been found so far.

comment:2 Changed 5 years ago by mithrandi

http://www.thinkgeek.com/ also has this defect, as of this posting.

comment:3 Changed 5 years ago by ivank

RFC 2616 section 19.3 Tolerant Applications:

The line terminator for message-header fields is the sequence CRLF.
However, we recommend that applications, when parsing such headers,
recognize a single LF as a line terminator and ignore the leading CR.

comment:4 Changed 5 years ago by ivank

(via http://news.ycombinator.com/item?id=705095 , which lists other HTTP clients that had to be modified to work with LF)

comment:5 Changed 4 years ago by ivank

Related bug for the old HTTP client: #2842

comment:6 Changed 4 years ago by jknight

http://www.thinkgeek.com/ no longer appears to have this problem.

But a new bad site was discovered: http://hdl.loc.gov/loc.uscongress/legislation.111s1448

(And I can't believe ycombinator is still so broken. Sigh.)

comment:7 Changed 4 years ago by exarkun

  • Keywords review added
  • Owner jknight deleted

The patch on #886 appears to be a more or less complete attempt at fixing this issue. Please review.

comment:8 Changed 4 years ago by jonathanj

  • Author set to jonathanj
  • Branch set to branches/newclient-bare-lf-3833

(In [30570]) Branching to 'newclient-bare-lf-3833'

comment:9 Changed 4 years ago by jonathanj

(In [30571]) Apply ivank's newclient patch for handling bare linefeeds in HTTP headers and status lines, refs #3833.

comment:10 Changed 4 years ago by jonathanj

  • Resolution set to fixed
  • Status changed from new to closed

(In [30573]) Merge newclient-bare-lf-3833: Handle HTTP headers delimited by bare LFs in Agent.

Author: ivank
Reviewer: jonathanj
Fixes: #3833

Some non-compliant web servers send HTTP responses delimited only by \n instead of \r\n. All modern web browsers support this and now Agent does too.

comment:11 Changed 4 years ago by <automation>

Note: See TracTickets for help on using tickets.