[Twisted-web] Client.agent - PartialDownloadError - cannot determine cause

[Glyph]

> On Apr 13, 2015, at 15:30, Chris Drane <csdrane at gmail.com> wrote:
> 
> I revisited the Stack Overflow post and it appears that I am able to receive HTTP responses properly now. Doing so required me to launch WireShark and copy a browser's actual headers. I also had to add a ContentDecoderAgent. 

This is a good idea, but it's also a bit of an accident.  Some sites may give you length prefixes with a content decoder, but there will still be some that provoke a PartialDownloadError no matter what you do.  There are edge cases in HTTP where you just cannot know if you have received everything the server sent, and many sites still operate that way.

> Specifically it was the lack of an Accept-Encoding that was causing the problem. I added ["gzip, deflate, sdch"] and it seemed to do the trick.

Hrm.  You have to set this header manually, even using a ContentDecoderAgent?  That sounds like a bug.

> I really do think that there should be an easier way to do this.

Absolutely.  From the very beginning, Agent was not really supposed to be a "high level" HTTP API, but those working on it sort of ran out of energy halfway through.  If you're writing applications today, you probably should use treq instead - https://github.com/twisted/treq <https://github.com/twisted/treq> - but longer-term the plan is to absorb treq or something very much like it into Twisted itself.

Looking through the ticket tracker, though, I see that the plan ... does not seem to be very well documented.  I can't find the "high level" ticket anywhere, and the closest thing I can find with just a few minutes of searching the tracker is this:

https://twistedmatrix.com/trac/ticket/3987#comment:29 <https://twistedmatrix.com/trac/ticket/3987#comment:29>

which closes a ticket about a "high level interface" by talking about a "mid-level API".  We clearly need a higher level API within Twisted itself.

> I also don't understand why the Agent couldn't have properly interpreted the initial response.

It did properly interpret the initial response.  You may or may not have received the whole body, and that's exactly what PartialDownloadError means.  If responses which might be the whole response OR might be the server breaking the connection on you are acceptable, handle that error and just treat the body that you have received so far as complete.  This is perfectly acceptable in many cases.

> Thanks for everyone's time.

Thanks for using Twisted!  Sorry that this experience was somewhat rocky.  I hope you'll stick around and help us improve it.

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-web/attachments/20150414/68a97e27/attachment-0001.html>