[Twisted-Python] Any Message Queue recommendations

Brett Viren bv at bnl.gov
Thu Jun 9 17:41:52 MDT 2005


Hi Ross,

Ross Jekel <ross at sourcelabs.com> writes:

> I'm writing several service processes that unfortunately need to
> integrate with Java, PHP, and potentially C/C++ programs (non-legacy).
> I've been searching for a message queuing layer that would support
> Python/C++/PHP and Java clients (JMS is not necessarily a requirement). 

This sounds a lot like what I asked recently regarding a network
distributed system for producing and consuming "alarms" (basically
just your "messages").

Consider this:

 - XML-RPC for communications.  XML-RPC implementations exist in all
 your target languages.  Twisted has addition support beyond the
 built-in Python XML-RPC.

 - Write a central proxy server speaking XML-RPC, (implemented in
 Twisted, of course!).  Every message comes through this proxy.

 - Message producers do simple XML-RPC query/response (method call) to
 send a message to the central proxy server.

 - Then, depending on how you want your message consumers to behave
 you can either:

  a) have them poll the server.  This can work in two ways.  Either
  the proxy returns any messages collected since some time, or you can
  have the query block until a message returns.  Or a mixture of both.

  b) have consumers register with the proxy and provide a callback
  XML-RPC url, the consumers then go into listen mode and the proxy
  pushes messages as they come in.

Option (a) is good because your consumers can be behind a firewall.
It's not so good as it introduces latency and makes the proxy more
complex as it must handle the buffering of messages.

Option (b) is simpler and introduces less latency between message
production and consumption but does require your consumers to be able
to listen on a port and receive incoming connections.  If there is a
firewall between the consumers and the proxy or if your local network
overlords don't like programs listening on ports this won't do.

You probably want to also allow for message levels and/or classes so
consumers can pick and choose what they receive.

And, if you go this route be sure to post your work so I can steal it
for my project!

Regards,
-Brett.




More information about the Twisted-Python mailing list