[Twisted-web] Re: URL length limits in twisted-web

[Jp Calderone]

On Thu, 25 Aug 2005 11:13:23 -0700 (PDT), Lenny G Arbage <alengarbage at yahoo.com> wrote:
>
>--- Tommi Virtanen wrote:
>>
>> There's this DoS-avoidance bit:
>>
>> class LineReceiver(protocol.Protocol,
>_PauseableMixin):
>>     ...
>>     MAX_LENGTH = 16384
>
>
>  Thanks, Tommi.  That appears to be it (16898 -
>len('http://') and the non "path" part of my URL = the
>more sane number of 16384).
>
>  The 'silent failure' feature doesn't seem quite
>right to me.

It isn't a silent failure.  The connection is dropped, as you notice.  The client is entirely capable of noticing that the server never responded to its request.  If no response is received, the correct thing to do is /not/ to assume everything worked perfectly.  Especially if the response was supposed to be a page.  twisted.web.client may well be buggy in its handling of this case (I've noticed several other bugs of this variety, though I'm not sure I've seen this one in particular).
> 
> [snip]
>
>  I changed LineReceiver.lineLengthReceived() as
>follows:
>
>def lineLengthExceeded(self, line):
>        """Called when the maximum line length has
>been reached.
>        Override if it needs to be dealt with in some
>special way.
>        """
>        #return self.transport.loseConnection()
>        self.transport.loseConnection()
>        raise error.ConnectionLost('Line length
>exceeded')
>
>  Which now does what I would expect (raises an
>exception if I try to send URLs that exceed the
>maximum), but I'm sure there are reasons not to raise
>an exception here (other classes that rely on
>LineReceiver, etc).

This just throws a random exception up into the reactor.  This is not allowed, but Twisted kindly catches it, logs it, and makes sure the protocol is not given any further data.  It's not an error condition you can reasonably deal with, it's just loud and ugly and obnoxious to make sure you notice your program is horribly broken and should be fixed.

>
>  What is the proper solution to this issue?

A better HTTP client.  Some people have been working in this area.  Perhaps they will grace us with a progress report.

Jp