[Twisted-Python] Design advice

benjamin.bertrand at lfv.se benjamin.bertrand at lfv.se
Fri Jan 14 09:56:53 EST 2011


Hi,

I'm trying to do a proof of concept and need some advice about the design.
We today have an external system sending messages to several of our tests systems (one specific connection per system).
We'd like to get our own server for more flexibility (easily setup connection to a new system, filtering, re-sending...).

The general idea so far is:
- A server receives all the messages from the external system, parses them and writes them to a database.
The server will be running as a daemon (should be on all the time).
- Clients can retrieve messages from the database and send them to test systems (one client per system). Clients should as well be able to send "live" data to the system (after some filtering on message content).
Clients are independent. They can be started/stopped depending on the need.

The server and clients use a specific protocol to communicate with the other system.
I already have implemented a server that receives messages from the external system. Parsing and writing them to a database shouldn't be too difficult.
I see how I can tell a client to retrieve specific messages from the database to send them.
What I'm not sure about is the "live" part (sending messages coming from the external system). Note that when I say "live", a small delay (up to 2-3 seconds) is not critical.
1) Each client could poll the database for change... They would need to poll it quite frequently, so don't think it's an option.
2) I read about being notified by a database on table changes (http://www.divillo.com/). That's much nicer than polling, but I don't think that's the best for my application.
3) I think a better solution would be to directly ask the server receiving the messages.
Not sure about the best way to do that:
3.1) The server could forward the raw data to all the clients and let the client do the filtering. Lot of unneeded data would be sent.
3.2) The server could do the filtering and send only the needed messages. Is using twisted's perspective broker a good solution for that?
4) Another idea?

In term of dimensioning, we can get around 200 messages per minute (max message length is 10000 bytes). Usually more around 20-30 messages per minute.
The number of clients is small (around 10).

Any advice?

Thanks

Benjamin



More information about the Twisted-Python mailing list