[Twisted-Python] Question regarding the working of twisted python with celery

Dinesh Kapoor dineshkapoor27 at gmail.com
Thu Mar 3 05:42:57 MST 2011


Thanks Reza and Jean-Paul for your response. Actually, I tried investigating
a bit more that whether the reactor is in different process than the calling
celery task, and it doesn't seem to be the case. The workflow that I am
assuming is as follows:

1. Celery creates a worker process and assigns a task to it, which is our
Python code.
2. That python code is calling into the twisted library where the reactor
exists, so that reactor is in the same process as our python code.
3. Our code opens up a TCP socket to another process which is NOT related to
celery or twisted (it connects to freeswitch process).
4. When celery is run with concurrency of 2, then I tried printing out the
process ids of calling process in the reactor code, and I am getting 2
different pids, so I am assuming there are two seperate copies of reactor in
those separate tasks.
5. One thing that I have think that's happening is that the reactor gets
stuck at polling, and then once it times out, the data is written on the
socket. Is that possible? If yes, how can I get around it?
6. Jean-Paul, how does Ampoule compares to Twisted?

Thanks,
Dinesh




On Wed, Mar 2, 2011 at 7:04 PM, <exarkun at twistedmatrix.com> wrote:

> On 09:42 am, dineshkapoor27 at gmail.com wrote:
> >Hi,
> >
> >  I have run into a weird performance issue while working with twisted
> >and
> >using celery to schedule tasks on it. Here is my setup:
> >
> >1. Celery schedules a task which makes a TCP connection to a server
> >running
> >FreeSwitch
> >2. Celery is running with concurrency = 2.
> >3. I have changed my code so that I call reactor.callfromthread for all
> >reactor based work.
> >4. A lot of times once I schedule the celery task, I get delays ranging
> >from
> >3 - 30 seconds when I am running with Celery concurrency =2.
> >5. If I reduce the celery concurrency to 1, then everything works
> >great!
> >6. Upon debugging it a little bit more, it seems that the
> >selectReactor's
> >doSelect is where the code is getting stuck on the select().
> >
> >I dont know how to get around the delay, and I dont know who is causing
> >that
> >delay exactly. If there is more info that is needed then please let me
> >know.
> >Everything works fine when the Celery worker process is 1, but delay
> >starts
> >happening invariably when celery's worker processes >=2.
>
> I think Reza Lotun explained what's causing this problem.
>
> If you haven't looked at it yet, you might want to see if Ampoule can
> help you out.
>
> Jean-Paul
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110303/fafbccb5/attachment.html>


More information about the Twisted-Python mailing list