[Twisted-Python] Need help for structured application development

Einar S. Idsø einar.twisted at norsk-esport.no
Sun Feb 21 06:56:38 EST 2010


On Fri, Feb 19, 2010 at 5:45 AM, Johann Borck
<johann.borck at densedata.com> wrote:
> I see. So if I get these requirements right you have 5-10 services with
> 1-30 instances of each, with following properties relevant to the task
> at hand:
>
> 1. those pollable by your existing program.
> 2. those incompatible with your existing program.
> 3. those that do not stream additional data.
> 4. those that do stream additional data.
>
> since sets  (1 and 2) and (3 and 4) are distinct, the combinations
> (1,3),(1,4),(2,3),(2,4) are possible.
>
> (1,3) the easy one, obviously you just need an object that polls the
> service using your program in intervals specific to the service.
> (1,4) Question: will the data you're interested in be collected by your
> existing program or by twisted? In the former case, it's basically the
> same as (1,3), in the latter you'll have to implement a protocol.
> (2,3) for these you'll need to implement a protocol class.
> (2,4) here you'll have to implement one or two protocols, depending on
> how the service is implemented.
>
> Is the above about correct? I think it would be a good idea to have an
> object OB that keeps references to all objects that gather data from the
> services, grouped by the type of service they're responsible for
> (defaultdict(list or dict) comes to mind). And then you'll probably
> either want factories that take care of handing newly created protocol
> instances over to OB or some instances (one or two per service in sets
> (2,3,4) ) of a multipurpose factory that can be initialized with the
> respective protocol and the information how to pass the created protocol
> over to OB, maybe  just a simple method that is able to distinguish the
> protocols by the interfaces they implement.
>
> One pitfall might be your polling program in case you're using
> t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous
> interface, so the worst that can happen is a stray process that doesn't
> return, but you still might want to consider implementing a
> ProcessProtocol
> (http://twistedmatrix.com/documents/current/core/howto/process.html)
> with a reasonable timeout, to be able to kill the spawned process in
> case it doesn't terminate. Thinking about it, it's the better solution
> anyway, because process protocols are just another type of protocol in
> twisted, and can be integrated consistently with the rest of your app.
>
> Another utility you might want is t.i.task.LoopingCall, for obvious
> reasons. Given your requirements something along these lines would be my
> approach, although I'd probably reimplement the polling thingy in
> twisted if it's not too complex :)
>
> hope that makes sense,
> Johann

Johann,

Thank you very much for your detailed response. It is greatly appreciated.

I think what you describe makes sense, and I hope I understand what
you mean and how I can implement it: I need to create a "suberobject"
OB which references all the poller-objects. I need to create one
object per detected server instance, which takes care of creating a
timed spawning of the external poller-process, and passing it along to
OB. For those servers that require streamlisteners, I really only need
one factory per server type, with the ablity to match streamed data
with polled data via the poller-object. Is this somewhat on the right
track?

I absolutely hate creating lots of code just to find that it was done
in the "wrong" way, structurally speaking, and that it requires a lot
of work to rewrite in the "correct way" to allow for further expansion
of the application. Your answers are very helpful in letting me avoid
spaghetti-code.

Cheers,
Einar



More information about the Twisted-Python mailing list