[Twisted-Python] Twisted web, giant-file POST forwarding and early bail-out.

Phil Mayers p.mayers at imperial.ac.uk
Tue Jul 9 02:41:11 MDT 2013


On 07/09/2013 09:04 AM, Rob Meijer wrote:
> Hi everyone,
>
> I'm working on what is just my second project using Twisted-Web, so I'm
> still a relative newbee on the subject.
>
> I'm working on a project that uses Twisted Web as a simple authorization
> proxy. All requests to my proxy contain an authorization-token and are
> either handled by the proxy, or are relayed to an other server. For all
> GET stuff and small POST stuff this is not a problem. When I want to
> process large POST requests however, I run into my limits of understanding
> how Twisted Web actually works.
>
> 1) I figured out that next to the 'process' in my request handler, I need
> to also overload handleContentChunk, parse the form body-parts in the
> first chunk myself and open a proxy connection (self.agent.request) if the
> authorization token checks out.
>
> 2) When it comes to appending the data received in handleContentChunk, and
> if needed throttling the client if the server couldn't keep up,  I can't
> figure out how to connect handleContentChunk and my self.agent.request
> instance.

You probably want to read up on the producer/consumer stuff in Twisted. 
In particular if you're using t.w.client.Agent, bodies in requests are 
supplied by an IBodyProducer.

http://twistedmatrix.com/documents/current/web/howto/client.html

Essentially, you need an IBodyProducer that maps to the incoming 
transport via request, which I guess would look something like this:

class RequestProducer(object):
     implements(IBodyProducer)

     def __init__(self, request):
         self.req = request

     def startProducing(self, consumer):
         self.d = defer.Deferred()
         self.consumer = consumer
         return d

     def pauseProducing(self):
         self.req.transport.pauseProducing()

     def stopProducing(self):
         # FIXME: what to do here...
         self.req.transport.loseConnection()

     def finish():
         self.d.callback(None)

...and you'll have code like this on the request object:

     def gotLength(self, length):
         self.bodyprod = RequestProducer(self)
         if length:
             self.bodyprod.length = length
         else:
             self.bodyprod.length = twisted.web.iweb.UNKNOWN_LENGTH
         self.out_req = self.agent.request(
             'GET', url, headers, bodyprod
         )

     def handleContentChunk(self, data):
         ...
         if data_to_be_forwarded:
             self.bodyprod.consumer.write(data)
         if some_done_condition:
             self.bodyprod.finish()
>
> 3) When the token does not check out, or the connection to the server
> fails, it remains a mystery to me how I should throw an error in such a
> way that it allows me to send a proper error message to the client, while

This is sort of a problem with HTTP. The client will probably keep 
sending the data.

The best you can do is write an HTTP error to the transport then throw 
the connection away, or blackhole all future content chunks.



More information about the Twisted-Python mailing list