[Twisted-Python] Implementing an event manager in twisted

Justin Warren daedalus at eigenmagic.com
Thu Apr 12 19:57:49 EDT 2007


On Thu, 2007-04-12 at 18:12 +0000, glyph at divmod.com wrote:
> On 12:26 am, daedalus at eigenmagic.com wrote:
> 
> >I have had an idea to implement an event manager to glue a few
> different
> >system components together, and I am wondering which pieces of
> twisted
> >would be best suited to the task, or even if I should attempt it at
> all.
> 
> Nothing about what you're suggesting sounds dangerous or weird.  In
> fact, it really sounds like what Twisted is already :).  If you want
> to write a layer over it to provide higher-level events, that's fine;
> most Twisted applications do something of the sort.  The one thing I'd
> caution you against is attempting to make something at the same level
> of abstraction as Twisted by building on top of Twisted.  That would
> just add complexity and overhead without any benefit.  So here are
> some things you've described which sound, to me, exactly like what
> Twisted already does:

Whew. :) I certainly don't plan to re-implement Twisted, since it
already exists and it's ace; I want something at a higher level. My aim
here is to make my existing application(s) even more modular so that
functionality can be added and extended in a simple, common way. This
principle has already provided excellent benefits in the past.

> >I have a desire to provide scheduler functionality (at time x, take
> >action y).
> 
> This sounds like reactor.callLater().  (Or if you're talking about
> persistent scheduling, axiom's IScheduler.schedule).

Yes, it will be persistent scheduling, so I'll check out IScheduler. "30
seconds have passed" would most likely be a reactor.callLater(30,
timePassed) thing though, yes, inside an EventProducer type object.

> >Data is received from an external entity by a listener.
> 
> This sounds like IProtocol's dataReceived method.

Yes, and I use it to great effect within the existing code. By listener,
I really mean a daemon process that listens, and there may be many of
them. Rather than have a single, monolithic twistd with many Services, I
think I would prefer a multi-process model so that each individual
process can be started/stopped independently, new ones can be written
that just talk to the event manager API, etc. In this way, the listener
process would purely listen for incoming data (eg: SNMP Traps), and then
talk to the event manager when it does.

In fact, now that I think about it, it would be neat if the event
manager API could allow third party programs to talk to the event
manager via a standard protocol, such as XML-RPC. Such programs could
then be written in the author's favourite language, which may not be
Python+twisted, and potentially operate on remote systems.

> >A program polls for data, succeeds and notifies the event manager of
> the
> >data that was polled. The event manager notifies a 'poll storage'
> object
> >that stores the data in the database. It could also notify other
> things.
> 
> In python, "notifying things" is just calling functions or methods; in
> the most advanced case, it's still just calling functions or methods
> which have been placed into a list.  Twisted has lots of such lists,
> and it doesn't seem worthwhile to me to unify them into a central
> structure.

True, though the more I think about it, I think I want an RPC mechanism
of some kind, and I believe Perspective Broker is such a beast? I
haven't really looked at it yet, but I've long been intrigued by what
people have been saying about it.

So it seems I want to add a veneer of RPC to my programs to allow them
to communicate at a distance. Imagine a poller process on one host, an
SNMP trap listener on another, a database storage process on still
another. This sort of modularity would provide excellent scalability.

Any suggestions for people's favourite ways of doing this with twisted?

> If it does make sense to you to unify them, you might want to look at
> Louie: http://pylouie.org/ .  I've never used it and I don't really
> get what it's for, but several folks that I know seem to like it a lot
> for gluing events together.

I'll check it out.

Thanks for the great responses so far, too. :)

-- 
Justin Warren <daedalus at eigenmagic.com>




More information about the Twisted-Python mailing list