[Twisted-Python] Help requested for twisted

Asha Joseph ashapattani at gmail.com
Tue Jul 2 05:29:52 MDT 2013


Hi ,

       In my case the rest api url calls looks like this

http://example.com/event/user_id=12345

  So here how to I implement the agent

for me every url has different different user_id's to be passes

Thanks and regards,
Asha







On Tue, Jul 2, 2013 at 8:59 AM, Asha Joseph <ashapattani at gmail.com> wrote:

> Hi Jean,
>    Thanks a lot for the quick response.
>
> All my requests are on the same server. or to the same domain.
>
> In that case do you suggest to use `twisted.web.client.Agent`
>
> I really don't require any responses from url that I send, it is fire and
> forget.
>
> thanks and regards,
> Asha
>
>
> On Mon, Jul 1, 2013 at 11:42 PM, <exarkun at twistedmatrix.com> wrote:
>
>> On 08:35 pm, ashapattani at gmail.com wrote:
>>
>>> Hi,
>>>
>>>  I am having a requirement on to send async some 300,000 or more rest api
>>> url calls..
>>>
>>
>> Are your HTTP requests all the same server?  Or are at least some of the
>> calls to the same server?  If so, if you use persistent connections
>> (`twisted.web.client.Agent` supports these) you'll remove a bunch of the
>> latency and overhead of those requests.
>>
>> Jean-Paul
>>
>>> When I work with the code for 150,000 it works fine .It takes around 8
>>> minutes.Is it possible to improve the performance
>>>
>>> When the count increased to some 200,000  ,I am getting the error.
>>>
>>>
>>>
>>> ------------------------------**------------------------------**
>>> ------------------------------**-------
>>>
>>> Traceback (most recent call last):
>>> Failure: twisted.internet.error.**TimeoutError: User timeout caused
>>> connection failure.
>>> Unhandled error in Deferred:
>>> Unhandled Error
>>> Traceback (most recent call last):
>>>
>>> ------------------------------**------------------------------**
>>> ------------------------------**-----
>>>
>>> As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM
>>>
>>>
>>> *Can you please suggest me how can I overcome this error *
>>> *
>>> *
>>> *Make the performance better*
>>>
>>>
>>> This is my piece of code where I have implemented twisted
>>> ------------------------------**------------------------------**
>>> ------------------------------**------
>>> from twisted.internet import defer, reactor, task
>>> from twisted.web.client import getPage
>>> import sys, argparse, csv, collections, time, datetime
>>>
>>> maxRun =32
>>> urls = ""
>>>
>>> def pageCallback(result):
>>>  print len(result)
>>>  return result
>>>
>>> def doWork():
>>>  for url in urls.split(','):
>>>    d = getPage(url)
>>>    d.addCallback(pageCallback)
>>>    yield d
>>>
>>> def finish(ign):
>>>  reactor.stop()
>>>
>>> def test():
>>>  deferreds = []
>>>  coop = task.Cooperator()
>>>  work = doWork()
>>>  for i in xrange(maxRun):
>>>    d = coop.coiterate(work)
>>>    deferreds.append(d)
>>>  dl = defer.DeferredList(deferreds)
>>>  dl.addCallback(finish)
>>>
>>>
>>> if __name__ == '__main__':
>>>
>>>   filename = sys.argv[1]
>>>   #I open the file ,read the files and create the url's
>>>
>>>
>>>      test()
>>>      reactor.run()
>>>
>>
>> ______________________________**_________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.**com <Twisted-Python at twistedmatrix.com>
>> http://twistedmatrix.com/cgi-**bin/mailman/listinfo/twisted-**python<http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20130702/b71d00f6/attachment-0002.html>


More information about the Twisted-Python mailing list