[Twisted-Python] Need help for structured application development
johann.borck at densedata.com
Thu Feb 18 23:45:28 EST 2010
Einar S. Idsø wrote:
> On Thu, Feb 18, 2010 at 11:36 AM, Johann Borck
> <johann.borck at densedata.com> wrote:
>> Einar S. Idsø wrote:
>>> [...] I guess I need a factory that launches server checker
>>> objects not as a result of a TCP/UDP connection, but as a result of
>>> new servers being detected.
>> How does this detection work? Are those servers based on twisted? Also,
>> do you control them (start/stop/configuration), do you have access to
>> the machines they're on, or are you detecting them over network only?
> Johann: Thanks for your response. I do control these services myself,
> and they run on our own hosts, but they are created and spawned via an
> external web interface which also has a Soap API. I do not plan on
> starting/stopping them via this application I'm developing. I just
> need to get the list of servers and start polling them. I get the list
> by calling the Soap webservice. It simply outputs a list of servers,
> and it is my job to query each and every one at specific intervals for
> I was thinking I would have the Soap-query run as a timed function
> every few minutes, and compare the new list with the old to work out
> any new/removed servers. Whenever a new server is identified, I will
> spawn a new object to deal with it, including, but not limited to,
> polling at given intervals. What kind of object should this be? A
> ClientFactory? Another kind of object that is already defined
> somewhere in Twisted? A completely custom object, built from scratch?
> I am leaning towards ClientFactory, but I am definitely not certain.
I see. So if I get these requirements right you have 5-10 services with
1-30 instances of each, with following properties relevant to the task
1. those pollable by your existing program.
2. those incompatible with your existing program.
3. those that do not stream additional data.
4. those that do stream additional data.
since sets (1 and 2) and (3 and 4) are distinct, the combinations
(1,3),(1,4),(2,3),(2,4) are possible.
(1,3) the easy one, obviously you just need an object that polls the
service using your program in intervals specific to the service.
(1,4) Question: will the data you're interested in be collected by your
existing program or by twisted? In the former case, it's basically the
same as (1,3), in the latter you'll have to implement a protocol.
(2,3) for these you'll need to implement a protocol class.
(2,4) here you'll have to implement one or two protocols, depending on
how the service is implemented.
Is the above about correct? I think it would be a good idea to have an
object OB that keeps references to all objects that gather data from the
services, grouped by the type of service they're responsible for
(defaultdict(list or dict) comes to mind). And then you'll probably
either want factories that take care of handing newly created protocol
instances over to OB or some instances (one or two per service in sets
(2,3,4) ) of a multipurpose factory that can be initialized with the
respective protocol and the information how to pass the created protocol
over to OB, maybe just a simple method that is able to distinguish the
protocols by the interfaces they implement.
One pitfall might be your polling program in case you're using
t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous
interface, so the worst that can happen is a stray process that doesn't
return, but you still might want to consider implementing a
with a reasonable timeout, to be able to kill the spawned process in
case it doesn't terminate. Thinking about it, it's the better solution
anyway, because process protocols are just another type of protocol in
twisted, and can be integrated consistently with the rest of your app.
Another utility you might want is t.i.task.LoopingCall, for obvious
reasons. Given your requirements something along these lines would be my
approach, although I'd probably reimplement the polling thingy in
twisted if it's not too complex :)
hope that makes sense,
More information about the Twisted-Python