[Twisted-web] Gzipped Response with web.client.Agent

[Vokdin Sergei]

10.08.10, 16:13, exarkun at twistedmatrix.com:

> On 9 Aug, 11:34 am, sergei.vokdin at yandex.ru wrote:
>  >Hello,
>  >
>  >I've putted together examples from the web (see below) to replace 
>  >getPage with web.client.Agent and it works fine. Now, I'd like to get 
>  >gzipped response, but I can't get gunzipping to work before returning 
>  >result.
>  >
>  >Thanks for help!
>  
>  What are you having trouble with?  Your code looks okay, more or less. 
>  If I were writing it, I'd try to make the gzip support more transparent, 
>  but the way you've done it seems like it should probably work.

Hi,

returned string should be wrapped into file object before passing to gzip module, that was the problem.

>  
>  A few simple things I notice that could cause problems, but won't 
>  necessarily...
>  
>    * Using repeated string concatenation to buffer the response is going 
>  to be extremely slow for responses of any significant size.

Nice catch, I'm forgetting this over and over again

>  
>    * In general, there's no guarantee you'll be able to decode the un- 
>  gzipped bytes using utf-8.  You could easily have downloaded a gzipped 
>  TIFF image.
>  
>    * Similarly, there's no guarantee that the un-gzipping will succeed if 
>  you got a truncated response (represented by the PotentialDataLoss 
>  failure).
>  
>    * The server might have sent back un-gzipped contents.  You have to 
>  check one of the response headers to see if it's appropriate to do the 
>  decompression.

Thanks, for hints. This was next step in implementation.


One issue I've still have is, how to set or implement timeout, so if remote host does not answer in some period of time, request would be cancelled.

Thanks a lot!