[Twisted-Python] Need help for structured application development

Einar S. Idsø einar.twisted at norsk-esport.no
Sun Feb 21 06:56:38 EST 2010

On Fri, Feb 19, 2010 at 5:45 AM, Johann Borck
<johann.borck at densedata.com> wrote:
> I see. So if I get these requirements right you have 5-10 services with
> 1-30 instances of each, with following properties relevant to the task
> at hand:
> 1. those pollable by your existing program.
> 2. those incompatible with your existing program.
> 3. those that do not stream additional data.
> 4. those that do stream additional data.
> since sets  (1 and 2) and (3 and 4) are distinct, the combinations
> (1,3),(1,4),(2,3),(2,4) are possible.
> (1,3) the easy one, obviously you just need an object that polls the
> service using your program in intervals specific to the service.
> (1,4) Question: will the data you're interested in be collected by your
> existing program or by twisted? In the former case, it's basically the
> same as (1,3), in the latter you'll have to implement a protocol.
> (2,3) for these you'll need to implement a protocol class.
> (2,4) here you'll have to implement one or two protocols, depending on
> how the service is implemented.
> Is the above about correct? I think it would be a good idea to have an
> object OB that keeps references to all objects that gather data from the
> services, grouped by the type of service they're responsible for
> (defaultdict(list or dict) comes to mind). And then you'll probably
> either want factories that take care of handing newly created protocol
> instances over to OB or some instances (one or two per service in sets
> (2,3,4) ) of a multipurpose factory that can be initialized with the
> respective protocol and the information how to pass the created protocol
> over to OB, maybe  just a simple method that is able to distinguish the
> protocols by the interfaces they implement.
> One pitfall might be your polling program in case you're using
> t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous
> interface, so the worst that can happen is a stray process that doesn't
> return, but you still might want to consider implementing a
> ProcessProtocol
> (http://twistedmatrix.com/documents/current/core/howto/process.html)
> with a reasonable timeout, to be able to kill the spawned process in
> case it doesn't terminate. Thinking about it, it's the better solution
> anyway, because process protocols are just another type of protocol in
> twisted, and can be integrated consistently with the rest of your app.
> Another utility you might want is t.i.task.LoopingCall, for obvious
> reasons. Given your requirements something along these lines would be my
> approach, although I'd probably reimplement the polling thingy in
> twisted if it's not too complex :)
> hope that makes sense,
> Johann


Thank you very much for your detailed response. It is greatly appreciated.

I think what you describe makes sense, and I hope I understand what
you mean and how I can implement it: I need to create a "suberobject"
OB which references all the poller-objects. I need to create one
object per detected server instance, which takes care of creating a
timed spawning of the external poller-process, and passing it along to
OB. For those servers that require streamlisteners, I really only need
one factory per server type, with the ablity to match streamed data
with polled data via the poller-object. Is this somewhat on the right

I absolutely hate creating lots of code just to find that it was done
in the "wrong" way, structurally speaking, and that it requires a lot
of work to rewrite in the "correct way" to allow for further expansion
of the application. Your answers are very helpful in letting me avoid


More information about the Twisted-Python mailing list