<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi ,</span><div style="font-family:arial,sans-serif;font-size:13px"><font face="arial, sans-serif"> </font></div><div style="font-family:arial,sans-serif;font-size:13px">
<font face="arial, sans-serif"> In my case the rest api url calls looks like this</font></div><div style="font-family:arial,sans-serif;font-size:13px"><font face="arial, sans-serif"><br></font></div><div style="font-family:arial,sans-serif;font-size:13px">
<font face="arial, sans-serif"><a href="http://example.com/event/user_id=12345" target="_blank">http://example.com/event/user_id=12345</a></font></div><div style="font-family:arial,sans-serif;font-size:13px"><font face="arial, sans-serif"><br>
</font></div><div style="font-family:arial,sans-serif;font-size:13px"><font face="arial, sans-serif"> So here how to I implement the agent</font></div><div style="font-family:arial,sans-serif;font-size:13px"><font face="arial, sans-serif"><br>
</font></div><div style="font-family:arial,sans-serif;font-size:13px"><font face="arial, sans-serif">for me every url has different different user_id's to be passes</font></div><div style="font-family:arial,sans-serif;font-size:13px">
<font face="arial, sans-serif"><br></font></div><div style="font-family:arial,sans-serif;font-size:13px"><font face="arial, sans-serif">Thanks and regards,</font></div><div style="font-family:arial,sans-serif;font-size:13px">
<font face="arial, sans-serif">Asha </font></div><div><font face="arial, sans-serif"><br></font></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 1, 2013 at 11:42 PM, <span dir="ltr"><<a href="mailto:exarkun@twistedmatrix.com" target="_blank">exarkun@twistedmatrix.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 08:35 pm, <a href="mailto:ashapattani@gmail.com" target="_blank">ashapattani@gmail.com</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I am having a requirement on to send async some 300,000 or more rest api<br>
url calls..<br>
</blockquote>
<br></div>
Are your HTTP requests all the same server? Or are at least some of the calls to the same server? If so, if you use persistent connections (`twisted.web.client.Agent` supports these) you'll remove a bunch of the latency and overhead of those requests.<br>
<br>
Jean-Paul<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
When I work with the code for 150,000 it works fine .It takes around 8<br>
minutes.Is it possible to improve the performance<br>
<br>
When the count increased to some 200,000 ,I am getting the error.<br>
<br>
<br>
<br>
------------------------------<u></u>------------------------------<u></u>------------------------------<u></u>-------<br>
<br>
Traceback (most recent call last):<br>
Failure: twisted.internet.error.<u></u>TimeoutError: User timeout caused<br>
connection failure.<br>
Unhandled error in Deferred:<br>
Unhandled Error<br>
Traceback (most recent call last):<br>
<br>
------------------------------<u></u>------------------------------<u></u>------------------------------<u></u>-----<br>
<br>
As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM<br>
<br>
<br></div>
*Can you please suggest me how can I overcome this error *<br>
*<br>
*<br>
*Make the performance better*<div><div class="h5"><br>
<br>
This is my piece of code where I have implemented twisted<br>
------------------------------<u></u>------------------------------<u></u>------------------------------<u></u>------<br>
from twisted.internet import defer, reactor, task<br>
from twisted.web.client import getPage<br>
import sys, argparse, csv, collections, time, datetime<br>
<br>
maxRun =32<br>
urls = ""<br>
<br>
def pageCallback(result):<br>
print len(result)<br>
return result<br>
<br>
def doWork():<br>
for url in urls.split(','):<br>
d = getPage(url)<br>
d.addCallback(pageCallback)<br>
yield d<br>
<br>
def finish(ign):<br>
reactor.stop()<br>
<br>
def test():<br>
deferreds = []<br>
coop = task.Cooperator()<br>
work = doWork()<br>
for i in xrange(maxRun):<br>
d = coop.coiterate(work)<br>
deferreds.append(d)<br>
dl = defer.DeferredList(deferreds)<br>
dl.addCallback(finish)<br>
<br>
<br>
if __name__ == '__main__':<br>
<br>
filename = sys.argv[1]<br>
#I open the file ,read the files and create the url's<br>
<br>
<br>
test()<br>
reactor.run()<br>
</div></div></blockquote>
<br>
______________________________<u></u>_________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com" target="_blank">Twisted-Python@twistedmatrix.<u></u>com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-<u></u>bin/mailman/listinfo/twisted-<u></u>python</a><br>
</blockquote></div><br></div>