[Twisted-Python] Application Design help - Concurrent but not Protocols based.

Minesh Patel minesh at gmail.com
Wed Jun 3 11:43:34 MDT 2009


> My Application Details:
>
> 1) I need to constantly monitor a particular directory for new files.
> 2) Whenever a new file is dropped; I read that file and get
> information on where to collect data from that is a) another machine b)
> machine2-different method c) database.
> 3) I collect data from those machines and store it.
>
> The data is huge and I need the three processes a, b, c to be
> non-blocking, and I can just do a function call like do_a(), do_b(),
> do_c() to perform them.

For this, since your data is huge and seems like it will need a lot of
CPU utilization, you might have to deferToThread, since Twisted runs
in a single thread and will block AFAIK.

d = threads.deferToThread(do_a, arg1_to_do_a, arg2...)
d.addCallback(do_a2, ...)

> For 1) to constantly monitor a particular directory for new files, I
> am doing something like this:
>
> while True:
>        check_for_new_files()
>
> http://paste.pocoo.org/show/120824/
>
> My Question: Can this be designed in way that looking for new files is
> also asynchronous activity?

This is how I am checking for new files via Twisted:

from twisted.internet import task
loopingFrequency = 10    # seconds between looping
task.LoopingCall(fun_a,  arg1_to_fun_a, arg2_to_fun_a).start(loopFrequency)

In fun_a I do the checking of new files


-- 
Thanks,
--Minesh




More information about the Twisted-Python mailing list