[Twisted-Python] Need help for structured application development

Einar S. Idsø einar.twisted at norsk-esport.no
Mon Feb 15 12:54:57 EST 2010


Hi,

I am planning to develop an application that will query a number of
services at given intervals, and Twisted seems like a very nice
platform to do so. However, because of little previous experience with
Twisted, especially with building proper applications, using
interfaces and using plugins, I could really need some pointers to get
me started.

In the following I will try to explain in broad terms what the
application will do, and humbly ask for input on how I may structure
it using factories, protocols etc.:

1. A number of more-or-less similar server applications (5-10) have a
number of concurrent instances/servers (1-30) each.
2. I need to acquire the status of each instance at given intervals.
These will typically range from every 20 seconds to every 3 minutes,
and depend on the which service the instance is based on.
3. A list of the server instances shall be acquired via an XML-query
every few minutes to pick up on new or removed instances.
4. An executable already exists which can be used to status-poll
almost all services using different parameters. I will be calling it
from my application to do the polling instead of recreating it in
Python.
5. I need to be able to create my own polling-protocol for at least
one of the service types which is incompatible with the executable.
6. Some services can not only be polled at intervals, but additionally
stream information that I am interested in listening to.
7. At a later point in time, I wish to be able to communicate with
IRC, MSN and other services/protocols using the queried and streamed
data.

My initial general thought is the following:
1. Design a class for each server application, where all classes
implement the same interface. The class defines how to deal with
server instances of its corresponding server application type. Classes
are created as Twisted-plugins.
2. For each new server instance that is discovered, create an object
which contains a timer and polls the server at specific intervals,
writing the result to a globally accessible data structure.
3. For server instances that stream data continuously in addition to
being polled at regular intervals, create a separate object which
listens to the server.

I believe I should somehow have a global data structure which contains
references to all the server instance objects, regardless of type, and
allows them to store data. This is sort of like the factory does for
protocols, except in my case I don't think I can use a regular factory
since I need both clients (to poll at intervals) and servers (to
listen to streaming data). I suppose I could use a ClientFactory for
the pollers and a Factory for the stream-listeners, and let them have
a common data-store? But is it possible to use a single ClientFactory
to spawn all poller-objects regardless of server type? Or do I need
one ClientFactory for each server type? Do I perhaps need one
ClientFactory per server instance? Or should I just ignore factories
all together since it is primarily the timer and asyncronicity of
Twisted I need?

I am quite certain there is a correct twistified way of handling this
sort of problem. I am just not sure what it is, and would sincerely
welcome all feedback on how this should be done. Feel free to point to
examples of applications that are similar to what I sketched above,
and tell me to study the code :)

Cheers,
Einar



More information about the Twisted-Python mailing list