[Twisted-Python] client, deferred and/or reactor hung on these 2 URL's
Jean-Paul Calderone
exarkun at divmod.com
Sat Jan 13 18:00:26 EST 2007
On Sat, 13 Jan 2007 16:38:04 -0600, Eric Mangold <teratorn at twistedmatrix.com> wrote:
>On Sat, 13 Jan 2007 08:00:11 -0600, Rick Graves <gravesricharde at yahoo.com>
>wrote:
>>Hello,
>>
>>I am new to twisted.
>>
>>I was using client.downloadPage and reactor.
>>
>>Twisted was hung on the following two for a whole hour:
>>
>>http://75i.net/
>>
>>http://www.theproxy.be/
>>
>>I had to break out with Ctrl-C. When I did that, they finally fired the
>>Errorback.
>
>I just wrote a test program that uses downloadPage() and it downloaded both
>those URLs fine.
>
>Are you still able to reproduce the problem? Does there seem to be any
>difference in downloading those URLs as part of a full run as opposed to
>individually? Do you have any proxies between you and those hosts? (Many
>ISPs have transparent proxies these days, *grrr*). If you can still
>reproduce then a traffic capture could be of use.
>
>In any case, the deferred returned from downloadPage probably shouldn't
>take > 1 hour to errback. It's certainly possible that downloadPage could
>need more than 1 hour to download a large file over a slow pipe, but that's
>not the case here.
Not necessarily. Networks have failures. It seems likely that there was a
transient error of some sort while downloading two URLs (out of several
hundred). Likely the connection would have eventually been lost, due to
TCP-level timeouts, but this would take longer than an hour. If such long
timeouts are not desirable for a particular application, you can set a lower
timeout by passing the timeout keyword argument to downloadPage. In these
cases, it may also be desirable to try again before ultimately reporting a
failure.
Jean-Paul
More information about the Twisted-Python
mailing list