[Twisted-Python] treq // Agent and Content-Length

[Donald Stufft]

I’m making a small proxy server in Twisted which is supposed to essentially
"mask" the original destination of an URL. Specifically it does the following:

1. Receives an HTTP request on /<hmac>/<url encoded url>
2. Verifies that the hmac in the URL matches the url decoded url
3. Makes an HTTP request to the url decoded url
4. Verify the response
   4a. Verify that the *on the wire* size is not greater than a set number of
       bytes, prior to downloading the body.
   4b. Verify that the content-type is acceptable prior to downloading the
       body.
5. Download the body of the response without modifying it (No Gzip decode)
6. Return the unmodified downloaded response as the resposne to the HTTP
   request in 1, as well as copying over several Response Headers.

This should be able to pass through ``Accept-Encoding`` from the request in
#1 to the request made in #3.

I've had a problem getting 4a to happen, no matter what I've tried if the
Content-Encoding is set to gzip it appears that the length of the response
is unknown. Additionally the "connection headers" are split out from the normal
headers, so I cannot seem to locate the original ``Content-Length`` header to
simply check that.

I got frustrated and implemented it using requests and that has worked OK. It
however uses deferToThread and I would like to get back to using the built
in tooling.

So my question is, how can I make Agent/treq do this?

The requests version of the code is here: https://gist.github.com/dstufft/7c8aac898de0ad359675
The treq versions of the code is here: https://gist.github.com/dstufft/0dffb12201d235f35bc7

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140303/ec6a2893/attachment.pgp>