Support for afterFoundGet in HTTPDownloader

[zimmer]

In twisted's current version HTTPDownloader http://twistedmatrix.com/trac/browser/trunk/twisted/web/client.py#L353 doesn't support afterFoundGet like HTTPClientFactory http://twistedmatrix.com/trac/browser/trunk/twisted/web/client.py#L220 does. Here is a way to do (patch?) that.

Replace HTTPDownloader's _ _init_ _ with this

class HTTPDownloader(HTTPClientFactory):
    """Download to a file."""

    protocol = HTTPPageDownloader
    value = None

    def __init__(self, url, fileOrName,
                 method='GET', postdata=None, headers=None,
                 agent="Twisted client", supportPartial=0,
                 timeout=0, cookies=None, followRedirect=1,
                 redirectLimit=20, afterFoundGet = False):
        self.requestedPartial = 0
        if isinstance(fileOrName, types.StringTypes):
            self.fileName = fileOrName
            self.file = None
            if supportPartial and os.path.exists(self.fileName):
                fileLength = os.path.getsize(self.fileName)
                if fileLength:
                    self.requestedPartial = fileLength
                    if headers == None:
                        headers = {}
                    headers["range"] = "bytes=%d-" % fileLength
        else:
            self.file = fileOrName
        HTTPClientFactory.__init__(
            self, url, method=method, postdata=postdata, headers=headers,
            agent=agent, timeout=timeout, cookies=cookies,
            followRedirect=followRedirect, redirectLimit=redirectLimit,
            afterFoundGet = afterFoundGet)

This code (above) adds a new argument to the init of HTTPDownloader (afterFoundGet) and then also passes it to the unbound init of HTTPClientFactory.

[magcius]

Patch.

[exarkun]

Thanks! Patch looks good to me. I'll apply it with just a couple minor tweaks:

1. add a .feature news fragment describing the new parameter
2. describe the behavior being tested more specifically in the test method docstring
3. add an extra blank line after the new test method, so there are two between it and the next test method

[exarkun]

(In [30712]) Add afterFoundGet parameter to twisted.web.client.downloadPage

Author: magcius Reviewer: exarkun Fixes: #4364

Add a parameter to downloadPage to allow controlling what type of request is issued after a FOUND response is received in response to a request. This afterFoundGet parameter works exactly the same way the getPage parameter of the same name works.