[Twisted-Python] Help requested for twisted

Asha Joseph ashapattani at gmail.com
Tue Jul 2 12:55:25 MDT 2013


Hi ,

       In my case the rest api url calls looks like this

http://example.com/event/user_id=12345

  So here how to I implement the agent

for me every url has different different user_id's to be passes

Thanks and regards,
Asha



On Mon, Jul 1, 2013 at 11:42 PM, <exarkun at twistedmatrix.com> wrote:

> On 08:35 pm, ashapattani at gmail.com wrote:
>
>> Hi,
>>
>>  I am having a requirement on to send async some 300,000 or more rest api
>> url calls..
>>
>
> Are your HTTP requests all the same server?  Or are at least some of the
> calls to the same server?  If so, if you use persistent connections
> (`twisted.web.client.Agent` supports these) you'll remove a bunch of the
> latency and overhead of those requests.
>
> Jean-Paul
>
>> When I work with the code for 150,000 it works fine .It takes around 8
>> minutes.Is it possible to improve the performance
>>
>> When the count increased to some 200,000  ,I am getting the error.
>>
>>
>>
>> ------------------------------**------------------------------**
>> ------------------------------**-------
>>
>> Traceback (most recent call last):
>> Failure: twisted.internet.error.**TimeoutError: User timeout caused
>> connection failure.
>> Unhandled error in Deferred:
>> Unhandled Error
>> Traceback (most recent call last):
>>
>> ------------------------------**------------------------------**
>> ------------------------------**-----
>>
>> As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM
>>
>>
>> *Can you please suggest me how can I overcome this error *
>> *
>> *
>> *Make the performance better*
>>
>>
>> This is my piece of code where I have implemented twisted
>> ------------------------------**------------------------------**
>> ------------------------------**------
>> from twisted.internet import defer, reactor, task
>> from twisted.web.client import getPage
>> import sys, argparse, csv, collections, time, datetime
>>
>> maxRun =32
>> urls = ""
>>
>> def pageCallback(result):
>>  print len(result)
>>  return result
>>
>> def doWork():
>>  for url in urls.split(','):
>>    d = getPage(url)
>>    d.addCallback(pageCallback)
>>    yield d
>>
>> def finish(ign):
>>  reactor.stop()
>>
>> def test():
>>  deferreds = []
>>  coop = task.Cooperator()
>>  work = doWork()
>>  for i in xrange(maxRun):
>>    d = coop.coiterate(work)
>>    deferreds.append(d)
>>  dl = defer.DeferredList(deferreds)
>>  dl.addCallback(finish)
>>
>>
>> if __name__ == '__main__':
>>
>>   filename = sys.argv[1]
>>   #I open the file ,read the files and create the url's
>>
>>
>>      test()
>>      reactor.run()
>>
>
> ______________________________**_________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.**com <Twisted-Python at twistedmatrix.com>
> http://twistedmatrix.com/cgi-**bin/mailman/listinfo/twisted-**python<http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20130702/c1b27842/attachment-0002.html>


More information about the Twisted-Python mailing list