[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests
Igor Katson
descentspb at gmail.com
Sat Oct 24 09:43:08 EDT 2009
exarkun at twistedmatrix.com wrote:
> On 01:20 pm, descentspb at gmail.com wrote:
>> Hello!
>>
>> I am a newbie in twisted, sorry if my question sounds awkward.
>>
>> I have written a pretty simple recursive page downloader, which parses
>> an html, extracts all the needed links from it, and starts dowloading
>> them. The links are the videofiles, so they are pretty large. The
>> problem is, that the downloader works TOO FAST :) I want to set
>> something like the global bandwidth limit or the maximum limit of
>> concurrently downloading files.
>>
>> I am using the twisted.web.client.downloadPage to download the files
>> and
>> using the Deferred, that it returns.
>> I can't understand how to make it still return a Deferred,
>> corresponding
>> to that file, but not start downloading right away, but instead start
>> downloading it on some kind of event (make a manger-like wrapper for
>> that function).
>>
>> So I want the code to still look simple like this:
>>
>> for link in links:
>> d = downloadPage_limited(link, filename)
>>
>> And the wrapper(function downloadPage_limited) will manage the amount
>> of
>> concurrent downloads, and will still return the Deferred, which will be
>> returned by twisted.web.client.downloadPage.
>>
>> Is my idea about a "wrapper" practical and what's the general way to
>> write it?
>> On which event is it better to decrement the counter of the amount
>> currently downloading files?
>
> Yes, that's a good idea.
>
> You might be able to use twisted.internet.defer.DeferredSemaphore to
> handle all of the counting for you. For example,
>
> from twisted.internet.defer import DeferredSemaphore
> from twisted.web.client import downloadPage
>
> class LimitedDownloader:
> def __init__(self, howMany):
> self._semaphore = DeferredSemaphore(howMany)
>
> def downloadPage(self, *a, **kw):
> return self._semaphore.run(downloadPage, *a, **kw)
>
> downloader = LimitedDownloader(3)
> downloader.downloadPage(...)
>
> In this example, DeferredSemaphore.run will only let 3 downloadPage
> calls run concurrently. If a 4th is attempted before any earlier ones
> finish, it won't actually be called until one of the earlier ones does
> finish, and then it will be called.
Thanks for quick and great help, Terry and Jean-Paul!
More information about the Twisted-web
mailing list