[Twisted-Python] client, deferred and/or reactor hung on these 2 URL's

Jean-Paul Calderone exarkun at divmod.com
Sat Jan 13 18:00:26 EST 2007


On Sat, 13 Jan 2007 16:38:04 -0600, Eric Mangold <teratorn at twistedmatrix.com> wrote:
>On Sat, 13 Jan 2007 08:00:11 -0600, Rick Graves <gravesricharde at yahoo.com> 
>wrote:
>>Hello,
>>
>>I am new to twisted.
>>
>>I was using client.downloadPage and reactor.
>>
>>Twisted was hung on the following two for a whole hour:
>>
>>http://75i.net/
>>
>>http://www.theproxy.be/
>>
>>I had to break out with Ctrl-C.  When I did that, they finally fired the 
>>Errorback.
>
>I just wrote a test program that uses downloadPage() and it downloaded  both 
>those URLs fine.
>
>Are you still able to reproduce the problem? Does there seem to be any 
>difference in downloading those URLs as part of a full run as opposed to 
>individually? Do you have any proxies between you and those hosts? (Many 
>ISPs have transparent proxies these days, *grrr*). If you can still 
>reproduce then a traffic capture could be of use.
>
>In any case, the deferred returned from downloadPage probably shouldn't 
>take > 1 hour to errback. It's certainly possible that downloadPage could 
>need more than 1 hour to download a large file over a slow pipe, but  that's 
>not the case here.

Not necessarily.  Networks have failures.  It seems likely that there was a
transient error of some sort while downloading two URLs (out of several
hundred).  Likely the connection would have eventually been lost, due to
TCP-level timeouts, but this would take longer than an hour.  If such long
timeouts are not desirable for a particular application, you can set a lower
timeout by passing the timeout keyword argument to downloadPage.  In these
cases, it may also be desirable to try again before ultimately reporting a
failure.

Jean-Paul




More information about the Twisted-Python mailing list