[Twisted-Python] Newbie question: Is this possible with twisted?

Reza Lotun rlotun at gmail.com
Wed Feb 24 15:40:44 EST 2010


Hi Tim,

> first of all: I've no experience with network-programming yet, but need to go in for creating a "simple" client-server application.

No problem.

> In the end I'd like to connect a web-form (django) with a twisted-server, which then should do some long-lasting calculations.
>
> So far I got a simple xmlrpc-server, which "defers" the calculation to a new thread and a client which sends the initial request to do the calculation. Though I have the problem that the client stays connected until the calculations are done.

Keep in mind you can run this "calculation server" as a separate
server-process and connect to it via loopback (localhost:SOME_PORT).
This makes it easier to put it on another machine one day without
changing the code.

Also, if you're doing a blocking calculation in Python (i.e. if it's
not a special C-module), then because of the Python Global Interpreter
Lock (GIL), you're not going to be able to paralllelise that
calculation effectively.

> Could someone tell me if it's somehow possible to disconnect the client right after sending the request to the server?

Well, it depends. Does the client need the result of that calculation?
Probably not, so what you can do is issue a job id back to the client,
and associate the job id to the result of that job. This is all very
doable in Twisted.

However, I'm going to suggest something different, which seems like a
better fit for what you're trying to do. Check out Celery
(http://celeryproject.org/) which is a distributed task queue. The
idea is that you set up a queue system, like RabbitMQ, which implement
a job queue via Celery. You write tasks as Python functions in a
module somewhere. When a client connects, you issue a task, and return
the task id back to the client. By and by the job is picked up by some
worker, and the result is stored in some persistent database, like
MySQL or Redis. Celery also has excellent integration with Django.

Hope that helps you out.

Cheers,
Reza

-- 
Reza Lotun
mobile: +44 (0)7521 310 763
email:  rlotun at gmail.com
work:   reza at tweetdeck.com
twitter: @rlotun



More information about the Twisted-Python mailing list