[Twisted-web] How can I change the HTTP request to avoid gzip

[Jean-Paul Calderone]

On Tue, 16 Dec 2008 15:57:18 +0200, Radu Dragusin <radudragusin at gmail.com> wrote:
>On Tue, Dec 16, 2008 at 2:20 PM, Jean-Paul Calderone <exarkun at divmod.com>wrote:
>
>> On Tue, 16 Dec 2008 05:31:45 +0200, Radu Dragusin <radudragusin at gmail.com>
>> wrote:
>>
> [snip]
>>>
>>> class WordCountProxyRequest(proxy.ProxyRequest):
>>>   protocols = {'http': WordCountProxyClientFactory}
>>>
>>>   def __init__(self, wordCounter, *args):
>>>       self.wordCounter = wordCounter
>>>       proxy.ProxyRequest.__init__(self, *args)
>>>
>>> *    def process(self):
>>>       proxy.ProxyRequest.process(self)
>>>       print "received_headers", proxy.ProxyRequest.getAllHeaders(self)*
>>
>>
>the print above prints:
>
>received_headers: {'accept-language': 'en-us,en;q=0.5', 'accept-encoding':
>'gzip,deflate', 'keep-alive': '300', 'accept':
>'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
>'user-agent': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4)
>Gecko/2008111318 Ubuntu/8.10 (intrepid) Firefox/3.0.4', 'accept-charset':
>'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'host': 'www.google.com', 'cookie':
>'PREF=ID=cfb3eb179de0c1e6:LD=en:NR=100:CR=2:TM=1228315308:LM=1229032156:GM=1:S=ImAuEufbnV6S7BAz;
>NID=17=lOVMiFLculcrfN-zUO7xxFTTUFzqQqaHOFHcG_BDmYFX8QKYbMoo7GrDoYH-8ASPBlVijG_Hstp7HSDQ_8WQexHPjwz6g_7ZVpBhwmh3vkKuO3jpf9dnzrnWthcW1mGh;
>S=photos_html=6ScUGfd699g4Xuuh0FeizA; TZ=-120', 'cache-control':
>'max-age=0', 'proxy-connection': 'keep-alive'}
>
>these are the values I want to modify, the 'accept-encoding', to be
>specific. How can I do it?

Request has a received_headers attribute which refers to a dictionary.  You
can delete the "accept" key from it.  Make sure you do it before you call
the base process method, though:

    def process(self):
        try:
            del self.received_headers['accept']
        except KeyError:
            pass
        proxy.ProxyRequest.process(self)

Or, in Twisted 8.2, you can use a slightly better headers API:

    def process(self):
        self.requestHeaders.removeHeader('accept')
        proxy.ProxyRequest.process(self)

Jean-Paul