[Twisted-Python] HTTPClientFactory's deferred never finishing download on .vcf (vcard file) link

[Steve Steiner (listsin)]

On Oct 10, 2009, at 7:34 AM, Reza Lotun wrote:

>> I'm not quite sure what you mean. Are you talking about web.client or
> web.server?

Web client.  I was having trouble with a particular URL which happened  
to be a download URL and wanted to see what the request and response  
objects looked like and was trying to find where I could examine  
them.  In the code I posted before, we were creating an  
HTTPClientFactory, then snagging the deferred.  I've simplified that  
by just using client.getPage().  The advantage of the  
HTTPClientFactory approach is that you can get at the response_headers.

> There is a new twisted.web client implementation in the works which  
> will hopefully be ready for trunk in the near future, and which  
> should provide nearly the same functionality offered by urllib2 (or  
> more).

Where might that be hiding?  I'd love to have something that does  
deferreds like Twisted and also implements all of urllib2.  I find the  
info() from urlopen() particularly helpful when debugging issues like  
this.

>> I finally figured out that it was a stupid string formatting bug in  
>> the code
>> that was being echoed by Twisted but swallowed by the "Grand  
>> Catcher of all
>> Things Exception" in Twisted..
>>
>> Once I found and fixed that, everything works fine -- until the  
>> next time.
>
> Exceptions shouldn't be swallowed by twisted. If you simply attach
> errBacks then you'll get the exception and full traceback. It really
> isn't that bad at all - I thought it looked crazy when I first
> started, but all the information is there to debug, etc.

Yes, the problem was in my mis-reading of the info in my errback.   
Unfortunately, the way the logging is set up in this particular code,  
logging.debug() returns 10 zillion times too much repetitive  
information and it makes it hard to wade through to find the relevant  
info (like the errback's output I was missing).

Thanks again for your help with this and I'd love to know more about  
this new urllib2 type module for Twisted.

Thanks,

S