[Twisted-Python] Question regarding the working of twisted python with celery

[Reza Lotun]

On Wed, Mar 2, 2011 at 9:42 AM, Dinesh Kapoor <dineshkapoor27 at gmail.com> wrote:
> 2. Celery is running with concurrency = 2.
> 3. I have changed my code so that I call reactor.callfromthread for all
> reactor based work.
> 4. A lot of times once I schedule the celery task, I get delays ranging from
> 3 - 30 seconds when I am running with Celery concurrency =2.
> 5. If I reduce the celery concurrency to 1, then everything works great!
> 6. Upon debugging it a little bit more, it seems that the selectReactor's
> doSelect is where the code is getting stuck on the select().

Celery makes synchronous blocking calls, and uses the multiprocessing
library. If I understand you correctly you are attempting to embed the
reactor in a celery worker process (your comment on callFromThread
seems to say this, at least). The problem is that with a concurrency
level of 2 the actual reactor could be running in another process.

Some things you can do:
- generate webhook tasks instead. Expose your twisted worker code via
and twisted.web interface - the celery tasks will simply POST to your
twisted.web endpoint and the actual worker code can be twistified.
- get celery worker code to make a TCP connection to your twisted
server. This is similar to the above idea, except HTTP isn't involved
and you have to come up with your own client-server protocol

Cheers,
Reza

-- 
Reza Lotun
mobile: +44 (0)7521 310 763
email:  rlotun at gmail.com
work:   reza at tweetdeck.com
twitter: @rlotun