[Twisted-Python] Question regarding the working of twisted python with celery

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Wed Mar 2 08:34:35 EST 2011


On 09:42 am, dineshkapoor27 at gmail.com wrote:
>Hi,
>
>  I have run into a weird performance issue while working with twisted 
>and
>using celery to schedule tasks on it. Here is my setup:
>
>1. Celery schedules a task which makes a TCP connection to a server 
>running
>FreeSwitch
>2. Celery is running with concurrency = 2.
>3. I have changed my code so that I call reactor.callfromthread for all
>reactor based work.
>4. A lot of times once I schedule the celery task, I get delays ranging 
>from
>3 - 30 seconds when I am running with Celery concurrency =2.
>5. If I reduce the celery concurrency to 1, then everything works 
>great!
>6. Upon debugging it a little bit more, it seems that the 
>selectReactor's
>doSelect is where the code is getting stuck on the select().
>
>I dont know how to get around the delay, and I dont know who is causing 
>that
>delay exactly. If there is more info that is needed then please let me 
>know.
>Everything works fine when the Celery worker process is 1, but delay 
>starts
>happening invariably when celery's worker processes >=2.

I think Reza Lotun explained what's causing this problem.

If you haven't looked at it yet, you might want to see if Ampoule can 
help you out.

Jean-Paul



More information about the Twisted-Python mailing list