[Twisted-Python] HTTP PUT a GET's streaming response with treq

[Phil Mayers]

On 04/05/17 16:30, Nagy, Attila wrote:

> I would like to use the simplest (and correct of course) solution.
> Juggling with buffering/data by hand seems even more risky to me.

The problem with the approach you've outlined is that it treats the 
transport (a private member) in ways that I suspect are invalid. In 
particular there's no handling of the length of the object or chunked 
encodings - I suspect what you're doing will only work on simple HTTP 
requests with no connection reuse.

I *think* if you use the t.web deliverBody response method, you get 
legal access to a transport (via an IProtocol you provide) and the API 
docs confirm that transport can be accessed and paused.

However if you look at:

https://github.com/twisted/twisted/blob/twisted-17.1.0/src/twisted/web/_newclient.py#L1069

...you'll see that deliverBody starts writing to the transport straight 
away, and you have no way to know if the consumer will have been 
provided to IBodyProducer at that point, or if it'll have been paused.

I think you inevitably will have to buffer some data, and a quick PoC of 
that code suggests to me that handling of the return deferred from 
startProducing is quite complex.


Perhaps someone more knowledgeable than I can chime in here?

> Your implementation seems to do things which twisted should do, which
> (also) doesn't seem to be right. :)
> It would be nice if this could be solved in an elegant way.

On that I completely agree. It could be that there's some simple idiom 
or built-in code for this that I'm unaware of.

> BTW (without reading it), I've tried your solution (thank you very much
> for it!).
> From a 1073741824 bytes source file, it transferred a 1056490639
> destination file.

Hey, I did say lightly tested ;o)

I think it's a bug in cleanly handling things at the end of the request.