[Twisted-Python] Help requested for twisted

[Asha Joseph]

Hi,

  I am having a requirement on to send async some 300,000 or more rest api
url calls..

When I work with the code for 150,000 it works fine .It takes around 8
minutes.Is it possible to improve the performance

When the count increased to some 200,000  ,I am getting the error.



-------------------------------------------------------------------------------------------------

Traceback (most recent call last):
Failure: twisted.internet.error.TimeoutError: User timeout caused
connection failure.
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):

-----------------------------------------------------------------------------------------------

As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM


*Can you please suggest me how can I overcome this error *
*
*
*Make the performance better*

This is my piece of code where I have implemented twisted
------------------------------------------------------------------------------------------------
from twisted.internet import defer, reactor, task
from twisted.web.client import getPage
import sys, argparse, csv, collections, time, datetime

maxRun =32
urls = ""

def pageCallback(result):
  print len(result)
  return result

def doWork():
  for url in urls.split(','):
    d = getPage(url)
    d.addCallback(pageCallback)
    yield d

def finish(ign):
  reactor.stop()

def test():
  deferreds = []
  coop = task.Cooperator()
  work = doWork()
  for i in xrange(maxRun):
    d = coop.coiterate(work)
    deferreds.append(d)
  dl = defer.DeferredList(deferreds)
  dl.addCallback(finish)


if __name__ == '__main__':

   filename = sys.argv[1]
   #I open the file ,read the files and create the url's


      test()
      reactor.run()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20130701/3d5de807/attachment.html>