[Twisted-Python] Re: Content Encoding : gzip ?

[anurag uniyal]

Thanks.

So it seems I have to write my own code, there in nothing like I can set some flag.

So if that is the case shouldn't I just be gzipping and unzipping data write and read to request object.

rgds
anurag


----- Original Message ----
From: David Bolen <db3l.net at gmail.com>
To: twisted-python at twistedmatrix.com
Sent: Friday, October 19, 2007 1:02:21 AM
Subject: [Twisted-Python] Re: Content Encoding : gzip ?


anurag uniyal <anuraguniyal at yahoo.com> writes:

> Is it possible to send and recieve compressed content using twisted.web?
>
> I have set Accept-encoding to gzip but it doesn't make any difference.
> Will I have to cater for this myself?

Yes, in twisted.web.  In twisted.web2, there's a filter that supports it.

Here's a small wrapper class I wrote recently that I've used to handle
this in twisted.web, with the goal of minimal changes to existing
resource code.  I use it by noticing, in the original request handler,
the ability to handle gzip encodings, and then wrap the original
request in this class (since there's no built-in filtering of the
output stream in twisted.web).

It's worked fine in my tests to date, and in limited production use,
but to be honest it hasn't seen large scale, long term use because my
original use case only yielded ~15% compression (multimedia files) and
using this negates the ability to know the full length in advance,
thus clients can't give projected download times.  (My files are large
enough that compressing twice is high overhead, and it wouldn't play
well with the wrapper approach anyway).

In any event, hopefully it'll be a useful starting point for you.

Here's the wrapper:

          - - - - - - - - - - - - - - - - - - - - - - - - -

import struct
import zlib

class GzipRequest(object):
    """Wrapper for a request that applies a gzip content encoding"""

    def __init__(self, request, compressLevel=6):
        self.request = request
        self.request.setHeader('Content-Encoding', 'gzip')
        # Borrowed from twisted.web2 gzip filter
        self.compress = zlib.compressobj(compressLevel, zlib.DEFLATED,
                                         -zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL,0)

    def __getattr__(self, attr):
        if 'request' in self.__dict__:
            return getattr(self.request, attr)
        else:
            raise AttributeError, attr

    def __setattr__(self, attr, value):
        if 'request' in self.__dict__:
            return setattr(self.request, attr, value)
        else:
            self.__dict__[attr] = value

    def write(self, data):
        if not self.request.startedWriting:
            self.crc = zlib.crc32('')
            self.size = self.csize = 0
            # XXX: Zap any length for now since we don't know final size
            if 'content-length' in self.request.headers:
                del self.request.headers['content-length']
            # Borrow header information from twisted.web2 gzip filter
            self.request.write('\037\213\010\000' '\0\0\0\0' '\002\377')

        self.crc = zlib.crc32(data, self.crc)
        self.size += len(data)
        cdata = self.compress.compress(data)
        self.csize += len(cdata)
        if cdata:
            self.request.write(cdata)
        elif self.request.producer:
            # Simulate another pull even though it hasn't really made it
            # out to the consumer yet.
            self.request.producer.resumeProducing()

    def finish(self):
        remain = self.compress.flush()
        self.csize += len(remain)
        if remain:
            self.request.write(remain)
        self.request.write(struct.pack('<LL',
                                       self.crc & 0xFFFFFFFFL,
                                       self.size & 0xFFFFFFFFL))
        self.request.finish()

          - - - - - - - - - - - - - - - - - - - - - - - - -

and here's a sample of using it.  This code is from one of my Resource
objects - if I were to use it more generally in my site I'd extract that
into a mix-in class of some sort - currently there was only one specific
resource (the multimedia files) I wanted to support it for:

    def render_GET(self, request):

        # (... argument validation ...)
        
        accept_encoding = request.getHeader('accept-encoding')
        if accept_encoding:
            encodings = accept_encoding.split(',')
            for encoding in encodings:
                name = encoding.split(';')[0].strip()
                if name == 'gzip':
                    request = GzipRequest(request)
                    break

        # At this point, use 'request' as normal


-- David


_______________________________________________
Twisted-Python mailing list
Twisted-Python at twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20071018/0b7a197b/attachment.htm