[Twisted-Python] Question regarding the working of twisted python with celery

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Wed Mar 2 08:34:35 EST 2011

On 09:42 am, dineshkapoor27 at gmail.com wrote:
>  I have run into a weird performance issue while working with twisted 
>using celery to schedule tasks on it. Here is my setup:
>1. Celery schedules a task which makes a TCP connection to a server 
>2. Celery is running with concurrency = 2.
>3. I have changed my code so that I call reactor.callfromthread for all
>reactor based work.
>4. A lot of times once I schedule the celery task, I get delays ranging 
>3 - 30 seconds when I am running with Celery concurrency =2.
>5. If I reduce the celery concurrency to 1, then everything works 
>6. Upon debugging it a little bit more, it seems that the 
>doSelect is where the code is getting stuck on the select().
>I dont know how to get around the delay, and I dont know who is causing 
>delay exactly. If there is more info that is needed then please let me 
>Everything works fine when the Celery worker process is 1, but delay 
>happening invariably when celery's worker processes >=2.

I think Reza Lotun explained what's causing this problem.

If you haven't looked at it yet, you might want to see if Ampoule can 
help you out.


More information about the Twisted-Python mailing list