[Twisted-Python] Pausing the transport in http.HTTPChannel.lineReceived after all headers are received

HawkOwl hawkowl at atleastfornow.net
Thu Mar 19 09:53:38 MDT 2015


> On 19 Mar 2015, at 23:26, Adi Roiban <adi at roiban.ro> wrote:
> 
> Hi,
> 
> My usecase is this:
> 
> When all headers of an HTTP request are received I want to ask a
> remote service if the
> request should be accepted and what to do with it.
> 
> Before I got the response for the remote service, I don't want to
> receive/read HTTP data.
> The response is returned via an deferred.
> 
> The HTTP data might be a 10G file which might be rejected.. for
> example due to quota limits.
> 
> Here is a gist with a short example and my quick and dirty fix:
> https://gist.github.com/adiroiban/621efeab47662a04cb20
> 
> ------
> 
> Do you think this is a valid use case and that  someone would be
> willing to review a patch to add such a feature?

Ah yes, the 100-Continue stuff! I mentioned it briefly in my talk at PyCon AU.

It would be a useful feature, and I looked into it a little bit, but I'm not entirely sure how it would be accessible to user code. Most web server frameworks (Django, Klein, Flask, et al) work on a "here's a full request, send a full response" (only Twisted/Klein can do more than send a whole response without hacks :) ) and fitting it into that sort of framework would be a bit confusing.

So, my question is, because this functionality is desirable, how would it be exposed to the user? Maybe if it did traversal prior to the whole request being completed, and called something like headers_POST() or headers_GET() similar to render_GET/POST/etc, which would give the user access to these headers. A Deferred firing with True unblocks the producer, a Deferred firing with False kills off the request.

> 
> -------
> 
> I first went on a wrong path as I was calling
> self.transport.pauseProducing() instead of self.pauseProducing() but
> while reading the code I found this:
> 
> In http.HTTPChannel  self.transport.producerPaused is confusing as it
> is always False.. and looking at the code from internet/abstract.py I
> can see that it is not touched by pauseProducing() ... how should
> producerPaused be used?
> 
> Is this a bug or a feature? Is there a ticket for it?
> 
> I saw that when you return a non-zero value from lineReceived() it
> will stop processing the remaining data but I could not find the
> documentation for this behaviour.
> 
> Should I continue searching for the documentation ? or file a ticket?

This functionality does not exist in Twisted, it just automatically sends the 100. (https://twistedmatrix.com/trac/ticket/4673) However, according to http://twisted-python.twistedmatrix.narkive.com/I9VkN3K3/twisted-web-giant-file-post-forwarding-and-early-bail-out , it probably won't work that great -- only some clients do the 100 continue stuff, and others will just keep shovelling data down the pipe, so only by closing the connection (and them probably disregarding your error message) will it work.

> 
> 
> Thanks!
> --
> Adi Roiban
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: </pipermail/twisted-python/attachments/20150319/021dde7a/attachment.sig>


More information about the Twisted-Python mailing list