[Twisted-Python] deferred classes

[Amaury Forgeot d'Arc]

Hello,

On Fri, Mar 7, 2008 at 2:05 PM,  <big_tyo at inf.its-sby.edu> wrote:
> hi can anyone explain to me how
>  defer.deferredlist,defer.deferredqueue,defer.deferredlock,and
>  defer.deferredsemaphore works?
>
>  an explanation with example will be very much help to me learning twisted

Deferreds are indeed a key feature in twisted.
First, did you read these two pages?
http://twistedmatrix.com/projects/core/documentation/howto/async.html
http://twistedmatrix.com/projects/core/documentation/howto/defer.html

(In summary: a Deferred is a promise that some result will come. To
get the result, you attach functions to the deferred; they will be
called when the result is available.)

Then the objects you list are easy to explain:
- defer.DeferredList is a deferred, which fires when all embedded
Deferreds have been fired.
- defer.DeferredQueue is a queue whose "get" method returns a Deferred
which fires when data is available.
- Similarly, defer.DeferredLock and defer.DeferredSemaphore are the
usual concurrency primitives, whose "acquire" method returns a
Deferred which fires when the acquisition is done.

Here is an example with DeferredSemaphore and DeferredList.
IMO the beauty of twisted is that the processing of one file is
described in a linear fashion: fetchOneFile() seems to run in
parallel.

===================================================

from twisted.internet import defer, reactor
from twisted.web.client import downloadPage
import os

# list of urls to retrieve
urls = [
    'http://xxx',
    'http://yyy',
    'http://zzz',
    'http://ttt',
]

# Only 2 simultaneous downloads
semaphore = defer.DeferredSemaphore(2)

def fetchOneFile(url):
    # Process one file:
    # - wait to get a a semaphore slot
    # - fetch the url
    # - release the semaphore

    d = semaphore.acquire()

    def fetchData(_):
        print "fetch", url
        return downloadPage(url, os.path.basename(url))
    d.addCallback(fetchData)

    def fetched(result):
        print "fetched", url, result
    d.addBoth(fetched)

    d.addBoth(lambda _:semaphore.release())

    return d


downloads = defer.DeferredList([fetchOneFile(url) for url in urls])
def end(result):
    reactor.stop()
downloads.addBoth(end)
reactor.run()


-- 
Amaury Forgeot d'Arc