[Twisted-web] Client.agent - PartialDownloadError - cannot determine cause

[Chris Drane]

I am new with Twisted so chances are that this is *not* a bug. If that is
the case, I would very much appreciate it if you help me understand what I
am doing incorrectly.

I have some very simple code to just download a particular page via HTTP.
The problem I'm running into is that I get PartialDownloadErrors for some
sites, while others load fine. And when I examine the HTTP response, it
looks like the page fully downloaded. The traceback and logging information
have been very unhelpful. I cannot determine what exactly is causing the
error.

import sys
from twisted.internet.task import react
from twisted.python.log import err, startLogging
from twisted.web.client import Agent, BrowserLikeRedirectAgent, readBody
from twisted.web.http_headers import Headers
from twisted.internet import reactor
from twisted.internet.ssl import ClientContextFactory

def cbBody(r):
    print "Response body:"
    print r

def cbRequest(response):
    print "Received response"
    d = readBody(response)
    d.addCallbacks(cbBody, err)
    return d

def err(e):
    try:
        e.raiseException()
    except Exception, err:
        pass
        # display HTTP response
        # print err.response
    e.printTraceback()
    sys.stderr.write(str(e))

def main(reactor):
    startLogging(sys.stdout)
    agent = BrowserLikeRedirectAgent(Agent(reactor))
    d = agent.request("GET", b"http://www.google.com",
            Headers({'User-Agent': ['Twisted Web Client Example']}),
            None)
    d.addCallbacks(cbRequest, err)
    return d

react(main)

2015-04-13 14:13:11-0400 [-] Log opened.
2015-04-13 14:13:11-0400 [-] Starting factory
<twisted.web.client._HTTP11ClientFactory instance at 0x10303f560>
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Received response
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Traceback (most
recent call last):
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Failure:
twisted.web.client.PartialDownloadError: 200 OK
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] [Failure instance:
Traceback (failure with no frames): <class
'twisted.web.client.PartialDownloadError'>: 200 OK
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Stopping factory
<twisted.web.client._HTTP11ClientFactory instance at 0x10303f560>
2015-04-13 14:13:11-0400 [-] Main loop terminated.

Thanks!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-web/attachments/20150413/916b2019/attachment.html>