[Twisted-Python] Application Design help - Concurrent but not Protocols based.

Senthil Kumaran orsenthil at gmail.com
Wed Jun 3 12:25:27 EDT 2009


Hello Twisted Developers/Users,

This is my first concurrent application design and my first trial with
twisted. I have read the documentation and understand where twisted
plays its part. Unfortunately, I could not directly relate it to my
requirements and hence, could not go forward with designing and
building on top using the examples as a reference.

I need your guidance in helping me design an application.

My Application Details:

1) I need to constantly monitor a particular directory for new files.
2) Whenever a new file is dropped; I read that file and get
information on where to collect data from that is a) another machine b)
machine2-different method c) database.
3) I collect data from those machines and store it.

The data is huge and I need the three processes a, b, c to be
non-blocking, and I can just do a function call like do_a(), do_b(),
do_c() to perform them.

For 1) to constantly monitor a particular directory for new files, I
am doing something like this:

while True:
        check_for_new_files()

http://paste.pocoo.org/show/120824/

My Question: Can this be designed in way that looking for new files is
also asynchronous activity? 

What will be the deferred in this case?

# my ideas:

- I might define a deferred as, whenever the contents of the directory
  is not matching the previous contents, return the new file which was
  added.
- I can then add a callback to read the newfile.


Now, after reading the contents, I will have to do a non-blocking call
to fetch data, either using fun_a, fun_b or fun_b. How should I
associate this requirement to deferred/callback pattern?

Any guidance would be helpful.

Thanks,
Senthil





More information about the Twisted-Python mailing list