[Twisted-Python] Help requested for twisted

[exarkun at twistedmatrix.com]

On 08:35 pm, ashapattani at gmail.com wrote:
>Hi,
>
>  I am having a requirement on to send async some 300,000 or more rest 
>api
>url calls..

Are your HTTP requests all the same server?  Or are at least some of the 
calls to the same server?  If so, if you use persistent connections 
(`twisted.web.client.Agent` supports these) you'll remove a bunch of the 
latency and overhead of those requests.

Jean-Paul
>When I work with the code for 150,000 it works fine .It takes around 8
>minutes.Is it possible to improve the performance
>
>When the count increased to some 200,000  ,I am getting the error.
>
>
>
>-------------------------------------------------------------------------------------------------
>
>Traceback (most recent call last):
>Failure: twisted.internet.error.TimeoutError: User timeout caused
>connection failure.
>Unhandled error in Deferred:
>Unhandled Error
>Traceback (most recent call last):
>
>-----------------------------------------------------------------------------------------------
>
>As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM
>
>
>*Can you please suggest me how can I overcome this error *
>*
>*
>*Make the performance better*
>
>This is my piece of code where I have implemented twisted
>------------------------------------------------------------------------------------------------
>from twisted.internet import defer, reactor, task
>from twisted.web.client import getPage
>import sys, argparse, csv, collections, time, datetime
>
>maxRun =32
>urls = ""
>
>def pageCallback(result):
>  print len(result)
>  return result
>
>def doWork():
>  for url in urls.split(','):
>    d = getPage(url)
>    d.addCallback(pageCallback)
>    yield d
>
>def finish(ign):
>  reactor.stop()
>
>def test():
>  deferreds = []
>  coop = task.Cooperator()
>  work = doWork()
>  for i in xrange(maxRun):
>    d = coop.coiterate(work)
>    deferreds.append(d)
>  dl = defer.DeferredList(deferreds)
>  dl.addCallback(finish)
>
>
>if __name__ == '__main__':
>
>   filename = sys.argv[1]
>   #I open the file ,read the files and create the url's
>
>
>      test()
>      reactor.run()