[Twisted-Python] Need help for structured application development

Johann Borck johann.borck at densedata.com
Thu Feb 18 23:45:28 EST 2010

Einar S. Idsø wrote:
> On Thu, Feb 18, 2010 at 11:36 AM, Johann Borck
> <johann.borck at densedata.com> wrote:
>> Einar S. Idsø wrote:
>>> [...]  I guess I need a factory that launches server checker
>>> objects not as a result of a TCP/UDP connection, but as a result of
>>> new servers being detected.
>> How does this detection work? Are those servers based on twisted? Also,
>> do you control them (start/stop/configuration), do you have access to
>> the machines they're on, or are you detecting them over network only?
> Johann: Thanks for your response. I do control these services myself,
> and they run on our own hosts, but they are created and spawned via an
> external web interface which also has a Soap API. I do not plan on
> starting/stopping them via this application I'm developing. I just
> need to get the list of servers and start polling them. I get the list
> by calling the Soap webservice. It simply outputs a list of servers,
> and it is my job to query each and every one at specific intervals for
> status.
> I was thinking I would have the Soap-query run as a timed function
> every few minutes, and compare the new list with the old to work out
> any new/removed servers. Whenever a new server is identified, I will
> spawn a new object to deal with it, including, but not limited to,
> polling at given intervals. What kind of object should this be? A
> ClientFactory? Another kind of object that is already defined
> somewhere in Twisted? A completely custom object, built from scratch?
> I am leaning towards ClientFactory, but I am definitely not certain.
I see. So if I get these requirements right you have 5-10 services with 
1-30 instances of each, with following properties relevant to the task 
at hand:

1. those pollable by your existing program.
2. those incompatible with your existing program.
3. those that do not stream additional data.
4. those that do stream additional data.

since sets  (1 and 2) and (3 and 4) are distinct, the combinations 
(1,3),(1,4),(2,3),(2,4) are possible.

(1,3) the easy one, obviously you just need an object that polls the 
service using your program in intervals specific to the service.
(1,4) Question: will the data you're interested in be collected by your 
existing program or by twisted? In the former case, it's basically the 
same as (1,3), in the latter you'll have to implement a protocol.
(2,3) for these you'll need to implement a protocol class.
(2,4) here you'll have to implement one or two protocols, depending on 
how the service is implemented.

Is the above about correct? I think it would be a good idea to have an 
object OB that keeps references to all objects that gather data from the 
services, grouped by the type of service they're responsible for 
(defaultdict(list or dict) comes to mind). And then you'll probably 
either want factories that take care of handing newly created protocol 
instances over to OB or some instances (one or two per service in sets 
(2,3,4) ) of a multipurpose factory that can be initialized with the 
respective protocol and the information how to pass the created protocol 
over to OB, maybe  just a simple method that is able to distinguish the 
protocols by the interfaces they implement.

One pitfall might be your polling program in case you're using 
t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous 
interface, so the worst that can happen is a stray process that doesn't 
return, but you still might want to consider implementing a 
with a reasonable timeout, to be able to kill the spawned process in 
case it doesn't terminate. Thinking about it, it's the better solution 
anyway, because process protocols are just another type of protocol in 
twisted, and can be integrated consistently with the rest of your app.

Another utility you might want is t.i.task.LoopingCall, for obvious 
reasons. Given your requirements something along these lines would be my 
approach, although I'd probably reimplement the polling thingy in 
twisted if it's not too complex :)

hope that makes sense,

More information about the Twisted-Python mailing list