Opened 7 years ago

Closed 6 years ago

#4364 defect closed fixed (fixed)

Support for afterFoundGet in HTTPDownloader

Reported by: zimmer Owned by:
Priority: normal Milestone:
Component: web Keywords: HTTPClientFactory, HTTPDownloader, afterFoundGet, 302 HTTP Redirect
Cc: Branch:
Author:

Description

In twisted's current version HTTPDownloader http://twistedmatrix.com/trac/browser/trunk/twisted/web/client.py#L353 doesn't support afterFoundGet like HTTPClientFactory http://twistedmatrix.com/trac/browser/trunk/twisted/web/client.py#L220 does. Here is a way to do (patch?) that.

Replace HTTPDownloader's _ _init_ _ with this

class HTTPDownloader(HTTPClientFactory):
    """Download to a file."""

    protocol = HTTPPageDownloader
    value = None

    def __init__(self, url, fileOrName,
                 method='GET', postdata=None, headers=None,
                 agent="Twisted client", supportPartial=0,
                 timeout=0, cookies=None, followRedirect=1,
                 redirectLimit=20, afterFoundGet = False):
        self.requestedPartial = 0
        if isinstance(fileOrName, types.StringTypes):
            self.fileName = fileOrName
            self.file = None
            if supportPartial and os.path.exists(self.fileName):
                fileLength = os.path.getsize(self.fileName)
                if fileLength:
                    self.requestedPartial = fileLength
                    if headers == None:
                        headers = {}
                    headers["range"] = "bytes=%d-" % fileLength
        else:
            self.file = fileOrName
        HTTPClientFactory.__init__(
            self, url, method=method, postdata=postdata, headers=headers,
            agent=agent, timeout=timeout, cookies=cookies,
            followRedirect=followRedirect, redirectLimit=redirectLimit,
            afterFoundGet = afterFoundGet)

This code (above) adds a new argument to the init of HTTPDownloader (afterFoundGet) and then also passes it to the unbound init of HTTPClientFactory.

Attachments (1)

after-found-get.diff (2.0 KB) - added by Jasper St. Pierre 6 years ago.
Patch.

Download all attachments as: .zip

Change History (5)

Changed 6 years ago by Jasper St. Pierre

Attachment: after-found-get.diff added

Patch.

comment:1 Changed 6 years ago by Jasper St. Pierre

Keywords: review added; removed

comment:2 Changed 6 years ago by Jean-Paul Calderone

Keywords: review removed
Owner: changed from jknight to Jean-Paul Calderone
Status: newassigned

Thanks! Patch looks good to me. I'll apply it with just a couple minor tweaks:

  1. add a .feature news fragment describing the new parameter
  2. describe the behavior being tested more specifically in the test method docstring
  3. add an extra blank line after the new test method, so there are two between it and the next test method

comment:3 Changed 6 years ago by Jean-Paul Calderone

Resolution: fixed
Status: assignedclosed

(In [30712]) Add afterFoundGet parameter to twisted.web.client.downloadPage

Author: magcius Reviewer: exarkun Fixes: #4364

Add a parameter to downloadPage to allow controlling what type of request is issued after a FOUND response is received in response to a request. This afterFoundGet parameter works exactly the same way the getPage parameter of the same name works.

comment:4 Changed 6 years ago by <automation>

Owner: Jean-Paul Calderone deleted
Note: See TracTickets for help on using tickets.