[Twisted-Python] Re: Content Encoding : gzip ?

[David Bolen]

anurag uniyal <anuraguniyal at yahoo.com> writes:

> Is it possible to send and recieve compressed content using twisted.web?
>
> I have set Accept-encoding to gzip but it doesn't make any difference.
> Will I have to cater for this myself?

Yes, in twisted.web.  In twisted.web2, there's a filter that supports it.

Here's a small wrapper class I wrote recently that I've used to handle
this in twisted.web, with the goal of minimal changes to existing
resource code.  I use it by noticing, in the original request handler,
the ability to handle gzip encodings, and then wrap the original
request in this class (since there's no built-in filtering of the
output stream in twisted.web).

It's worked fine in my tests to date, and in limited production use,
but to be honest it hasn't seen large scale, long term use because my
original use case only yielded ~15% compression (multimedia files) and
using this negates the ability to know the full length in advance,
thus clients can't give projected download times.  (My files are large
enough that compressing twice is high overhead, and it wouldn't play
well with the wrapper approach anyway).

In any event, hopefully it'll be a useful starting point for you.

Here's the wrapper:

          - - - - - - - - - - - - - - - - - - - - - - - - -

import struct
import zlib

class GzipRequest(object):
    """Wrapper for a request that applies a gzip content encoding"""

    def __init__(self, request, compressLevel=6):
        self.request = request
        self.request.setHeader('Content-Encoding', 'gzip')
        # Borrowed from twisted.web2 gzip filter
        self.compress = zlib.compressobj(compressLevel, zlib.DEFLATED,
                                         -zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL,0)

    def __getattr__(self, attr):
        if 'request' in self.__dict__:
            return getattr(self.request, attr)
        else:
            raise AttributeError, attr

    def __setattr__(self, attr, value):
        if 'request' in self.__dict__:
            return setattr(self.request, attr, value)
        else:
            self.__dict__[attr] = value

    def write(self, data):
        if not self.request.startedWriting:
            self.crc = zlib.crc32('')
            self.size = self.csize = 0
            # XXX: Zap any length for now since we don't know final size
            if 'content-length' in self.request.headers:
                del self.request.headers['content-length']
            # Borrow header information from twisted.web2 gzip filter
            self.request.write('\037\213\010\000' '\0\0\0\0' '\002\377')

        self.crc = zlib.crc32(data, self.crc)
        self.size += len(data)
        cdata = self.compress.compress(data)
        self.csize += len(cdata)
        if cdata:
            self.request.write(cdata)
        elif self.request.producer:
            # Simulate another pull even though it hasn't really made it
            # out to the consumer yet.
            self.request.producer.resumeProducing()

    def finish(self):
        remain = self.compress.flush()
        self.csize += len(remain)
        if remain:
            self.request.write(remain)
        self.request.write(struct.pack('<LL',
                                       self.crc & 0xFFFFFFFFL,
                                       self.size & 0xFFFFFFFFL))
        self.request.finish()

          - - - - - - - - - - - - - - - - - - - - - - - - -

and here's a sample of using it.  This code is from one of my Resource
objects - if I were to use it more generally in my site I'd extract that
into a mix-in class of some sort - currently there was only one specific
resource (the multimedia files) I wanted to support it for:

    def render_GET(self, request):

        # (... argument validation ...)
        
        accept_encoding = request.getHeader('accept-encoding')
        if accept_encoding:
            encodings = accept_encoding.split(',')
            for encoding in encodings:
                name = encoding.split(';')[0].strip()
                if name == 'gzip':
                    request = GzipRequest(request)
                    break

        # At this point, use 'request' as normal


-- David