[Twisted-Python] Twisted web, giant-file POST forwarding and early bail-out.

[Rob Meijer]

Hi everyone,

I'm working on what is just my second project using Twisted-Web, so I'm
still a relative newbee on the subject.

I'm working on a project that uses Twisted Web as a simple authorization
proxy. All requests to my proxy contain an authorization-token and are
either handled by the proxy, or are relayed to an other server. For all
GET stuff and small POST stuff this is not a problem. When I want to
process large POST requests however, I run into my limits of understanding
how Twisted Web actually works.

1) I figured out that next to the 'process' in my request handler, I need
to also overload handleContentChunk, parse the form body-parts in the
first chunk myself and open a proxy connection (self.agent.request) if the
authorization token checks out.

2) When it comes to appending the data received in handleContentChunk, and
if needed throttling the client if the server couldn't keep up,  I can't
figure out how to connect handleContentChunk and my self.agent.request
instance.

3) When the token does not check out, or the connection to the server
fails, it remains a mystery to me how I should throw an error in such a
way that it allows me to send a proper error message to the client, while
not having to first accept the whole large file. That is, it seems rather
silly that I would know things failed after the first POST body chunk, but
would have to wait for and accept hundreds of megabytes or maybe even a
few gigabytes of post data before I can notify the client that something
went wrong.


It seems I am either missing something blindingly obvious or Twisted Web
simply isn't meant to be used this way. I hope someone can give me some
directions how to make this giant-file-post forwarding and early-bail-out
scenario working with Twisted Web.

T.I.A.

Rob Meijer