[Twisted-Python] Application Design help - Concurrent but not Protocols based.

[Senthil Kumaran]

On Wed, Jun 03, 2009 at 10:43:34AM -0700, Minesh Patel wrote:

> For this, since your data is huge and seems like it will need a lot of
> CPU utilization, you might have to deferToThread, since Twisted runs
> in a single thread and will block AFAIK.
> 
> d = threads.deferToThread(do_a, arg1_to_do_a, arg2...)
> d.addCallback(do_a2, ...)
> 

I delayed and forgot to respond to this thread. Sorry for that.
Actually, I had asked the question on doing asynchronous monitoring of
new file arrival and then based on the file contents, doing tasks
a,b,c asynchronously.

Thank you for your responses. I got a number of pointers to look at.

* As a newbie to this world, I just started studying further.

I realized that in my application, I have the control form the client
on when I can notify that a particular file can be looked at.
So, for the time being, I just skipped the inotify linux system
related calls ( BTW, Twsited + inotify works great) and when ahead
with just implementing a Twisted a Server which listens to a
particular port and when receives data, will either read the file or
parse the data to do some activity (like fetch logs from database).

threads.deferToThread(func,[args]) is what I am using.

http://paste.pocoo.org/show/122155/


I am still working out more details as doing this
threads.deferToThread calls effectively, because there could be number
of clients parallely requesting it and the data from my query
(internal function) could be so big that it might take 30 minutes to
service a request.

* Another thing is when using something like 
"from datasources import query"

I could not make it to application (.tac to run it from twistd), it
could nto find datasources in its PYTHONPATH. Got to figure out how to
do it, which I might pretty soon.


Thanks again,
Senthil