Thanks Reza and Jean-Paul for your response. Actually, I tried investigating a bit more that whether the reactor is in different process than the calling celery task, and it doesn't seem to be the case. The workflow that I am assuming is as follows:<br>
<br>1. Celery creates a worker process and assigns a task to it, which is our Python code.<br>2. That python code is calling into the twisted library where the reactor exists, so that reactor is in the same process as our python code.<br>
3. Our code opens up a TCP socket to another process which is NOT related to celery or twisted (it connects to freeswitch process).<br>4. When celery is run with concurrency of 2, then I tried printing out the process ids of calling process in the reactor code, and I am getting 2 different pids, so I am assuming there are two seperate copies of reactor in those separate tasks.<br>
5. One thing that I have think that's happening is that the reactor gets stuck at polling, and then once it times out, the data is written on the socket. Is that possible? If yes, how can I get around it?<br>6. Jean-Paul, how does Ampoule compares to Twisted?<br>
<br>Thanks,<br>Dinesh<br><br><br><br>
<br><div class="gmail_quote">On Wed, Mar 2, 2011 at 7:04 PM, <span dir="ltr"><<a href="mailto:exarkun@twistedmatrix.com" target="_blank">exarkun@twistedmatrix.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div>On 09:42 am, <a href="mailto:dineshkapoor27@gmail.com" target="_blank">dineshkapoor27@gmail.com</a> wrote:<br>
>Hi,<br>
><br>
> I have run into a weird performance issue while working with twisted<br>
>and<br>
>using celery to schedule tasks on it. Here is my setup:<br>
><br>
>1. Celery schedules a task which makes a TCP connection to a server<br>
>running<br>
>FreeSwitch<br>
>2. Celery is running with concurrency = 2.<br>
>3. I have changed my code so that I call reactor.callfromthread for all<br>
>reactor based work.<br>
>4. A lot of times once I schedule the celery task, I get delays ranging<br>
>from<br>
>3 - 30 seconds when I am running with Celery concurrency =2.<br>
>5. If I reduce the celery concurrency to 1, then everything works<br>
>great!<br>
>6. Upon debugging it a little bit more, it seems that the<br>
>selectReactor's<br>
>doSelect is where the code is getting stuck on the select().<br>
><br>
>I dont know how to get around the delay, and I dont know who is causing<br>
>that<br>
>delay exactly. If there is more info that is needed then please let me<br>
>know.<br>
>Everything works fine when the Celery worker process is 1, but delay<br>
>starts<br>
>happening invariably when celery's worker processes >=2.<br>
<br>
</div></div>I think Reza Lotun explained what's causing this problem.<br>
<br>
If you haven't looked at it yet, you might want to see if Ampoule can<br>
help you out.<br>
<br>
Jean-Paul<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com" target="_blank">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
</blockquote></div><br>