[Twisted-Python] web client FileBodyProducer - transfer encoding

[Kevin Mcintyre]

"It is shaped the same.  The reason you're seeing the error is due to the
issue I pointed out above."

Just to be clear you mean that proxies can reshape the call in flight?



On Mon, Jan 30, 2017 at 6:01 PM, Glyph Lefkowitz <glyph at twistedmatrix.com>
wrote:

>
> On Jan 30, 2017, at 19:44, Kevin Mcintyre <kebin70 at gmail.com> wrote:
>
> On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone <
> exarkun at twistedmatrix.com> wrote:
>
>> On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz <glyph at twistedmatrix.com
>> > wrote:
>>
>>> Gotcha.  I guess what I meant was that you shouldn't care about this at
>>> the application level, but you're talking about an operational concern, not
>>> an application-level concern.
>>>
>>> Perhaps this should be a tunable on Agent somehow.  Can you file a
>>> ticket?
>>>
>>
>>
>> Well... It *is* sort of tuneable, as you implied earlier.
>>
>> If you pass an IBodyProducer with a non-None length, Agent will send a
>> Content-Length header - not use chunked Transfer-Encoding.
>> FileBodyProducer doesn't know how to determine the length of a StringIO, so
>> you get chunked with this example.
>>
>
> Not quite true: FileBodyProducer *does* know how to compute the length of
> a StringIO, via seek():
>
>
> >>> from twisted.web.client import FileBodyProducer
> >>> from io import BytesIO
> >>> from cStringIO import StringIO
> >>> FileBodyProducer(BytesIO(b"some bytes")).length
> 10L
> >>> FileBodyProducer(StringIO(b"some bytes")).length
> 10
>
>
> If you write the JSON to a regular file and FileBodyProducer(open(the
>> file)) you'll get a Content-Length request.  You could also write a new
>> (trivial) IBodyProducer that does know how to compute the length of a
>> StringIO.
>>
>> The documentation doesn't exactly spell this out - but the only reason
>> `length` is part of the interface is to be able to generate the
>> Content-Length header.
>>
>> What would the ticket be?  Expanded documentation to make this behavior
>> into an explicit guarantee of the interface?  A new toggle somewhere to
>> force Agent.request into one mode or the other (regardless of the
>> performance consequences)?
>>
>
> Rather than "regardless" of the consequences, perhaps just a maximum body
> size.  If we made it an explicit guarantee in Agent, perhaps the interface
> change would not be in '.request' (which, as a formal interface used both
> inside and outside of Twisted, is fairly fixed), but rather a new
> `NonChunkedBodyProducer` concrete class that would implement IBodyProducer
> in terms of another IBodyProducer which either does or doesn't have a
> `length`.  (Or a function that does same, always returning its argument if
> `length` is already set...)
>
> And that stuff you said about proxies before was true ... So even if you
>> can control this in Agent, you're still not *guaranteed* the server will
>> see what you send.
>>
>
> Thanks - that's exactly what I was looking for.
>
> *But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I
> still see a content-length in the request header as it's received by
> twisted.  Am I correct to assume the request is agnostic...meaning it's
> shaped the same for twisted as it is for apache.
>
>
> It is shaped the same.  The reason you're seeing the error is due to the
> issue I pointed out above.
>
> Headers({'host': ['localhost:8080'], 'connection': ['close'],
> 'content-length': ['671'], 'user-agent': ['PassengerTest']})
>
>
> -glyph
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170130/df070132/attachment-0002.html>