[Twisted-Python] web client FileBodyProducer - transfer encoding

[Glyph Lefkowitz]

> On Jan 30, 2017, at 19:44, Kevin Mcintyre <kebin70 at gmail.com> wrote:
> On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone <exarkun at twistedmatrix.com <mailto:exarkun at twistedmatrix.com>> wrote:
> 
> On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> Gotcha.  I guess what I meant was that you shouldn't care about this at the application level, but you're talking about an operational concern, not an application-level concern.
> 
> Perhaps this should be a tunable on Agent somehow.  Can you file a ticket?
> 
> 
> Well... It is sort of tuneable, as you implied earlier.
> 
> If you pass an IBodyProducer with a non-None length, Agent will send a Content-Length header - not use chunked Transfer-Encoding.  FileBodyProducer doesn't know how to determine the length of a StringIO, so you get chunked with this example.

Not quite true: FileBodyProducer does know how to compute the length of a StringIO, via seek():

>>> from twisted.web.client import FileBodyProducer
>>> from io import BytesIO
>>> from cStringIO import StringIO
>>> FileBodyProducer(BytesIO(b"some bytes")).length
10L
>>> FileBodyProducer(StringIO(b"some bytes")).length
10

> If you write the JSON to a regular file and FileBodyProducer(open(the file)) you'll get a Content-Length request.  You could also write a new (trivial) IBodyProducer that does know how to compute the length of a StringIO.
> 
> The documentation doesn't exactly spell this out - but the only reason `length` is part of the interface is to be able to generate the Content-Length header.
> 
> What would the ticket be?  Expanded documentation to make this behavior into an explicit guarantee of the interface?  A new toggle somewhere to force Agent.request into one mode or the other (regardless of the performance consequences)?

Rather than "regardless" of the consequences, perhaps just a maximum body size.  If we made it an explicit guarantee in Agent, perhaps the interface change would not be in '.request' (which, as a formal interface used both inside and outside of Twisted, is fairly fixed), but rather a new `NonChunkedBodyProducer` concrete class that would implement IBodyProducer in terms of another IBodyProducer which either does or doesn't have a `length`.  (Or a function that does same, always returning its argument if `length` is already set...)

> And that stuff you said about proxies before was true ... So even if you can control this in Agent, you're still not guaranteed the server will see what you send.

> Thanks - that's exactly what I was looking for.
> 
> But note when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I still see a content-length in the request header as it's received by twisted.  Am I correct to assume the request is agnostic...meaning it's shaped the same for twisted as it is for apache.

It is shaped the same.  The reason you're seeing the error is due to the issue I pointed out above.
> Headers({'host': ['localhost:8080'], 'connection': ['close'], 'content-length': ['671'], 'user-agent': ['PassengerTest']})
> 

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170130/83115068/attachment-0002.html>