[Twisted-Python] Application Design help - Concurrent but not Protocols based.
minesh at gmail.com
Wed Jun 3 13:43:34 EDT 2009
> My Application Details:
> 1) I need to constantly monitor a particular directory for new files.
> 2) Whenever a new file is dropped; I read that file and get
> information on where to collect data from that is a) another machine b)
> machine2-different method c) database.
> 3) I collect data from those machines and store it.
> The data is huge and I need the three processes a, b, c to be
> non-blocking, and I can just do a function call like do_a(), do_b(),
> do_c() to perform them.
For this, since your data is huge and seems like it will need a lot of
CPU utilization, you might have to deferToThread, since Twisted runs
in a single thread and will block AFAIK.
d = threads.deferToThread(do_a, arg1_to_do_a, arg2...)
> For 1) to constantly monitor a particular directory for new files, I
> am doing something like this:
> while True:
> My Question: Can this be designed in way that looking for new files is
> also asynchronous activity?
This is how I am checking for new files via Twisted:
from twisted.internet import task
loopingFrequency = 10 # seconds between looping
task.LoopingCall(fun_a, arg1_to_fun_a, arg2_to_fun_a).start(loopFrequency)
In fun_a I do the checking of new files
More information about the Twisted-Python