[Twisted-Python] AMQP listening and user-facing daemon

[Justin Chiu]

Hi all,

*I posted this to python-list and tutor-list and received no replies. 
Any advice would be much appreciated. Thank you.*

What is the best approach to writing a concurrent daemon that can 
execute callbacks for different types of events (AMQP messages, parsed 
output of a subprocess, HTTP requests)?

I am considering [twisted][1], the built-in [threading][2] module, and 
[greenlet][3]. I must admit that I am very unfamiliar with concurrent 
programming and Python programming in general (formerly a data analysis 
driven procedural programmer). Any resources on threaded/concurrent 
programming (specifically daemons...not just multi-threading a single 
task) would be much appreciated.

Thank you.

Details:

1) Listens into AMQP messaging queues and executes callbacks when 
messages arrive.
Example: Immediately after startup, the daemon continuously listens to 
the [Openstack Notifications messaging queue][4]. When a virtual machine 
is launched, a notification is generated by Openstack with the hostname, 
IP address, etc. The daemon should read this message and write some info 
to a log (or POST the info to a server, or notify the user...something 
simple).

2) Parse the output of a subprocess and execute callbacks based on the 
output.
Example: Every 30 seconds, a system command "[qstat][5]" is run to query 
a job resource manager (e.g. TORQUE). Similar callbacks to 1).

3) Receive requests from a user and process them. I think this will be 
via WSGI HTTP.
Example: User submits an XML template with virtual machine templates. 
The daemon does some simple XML parsing and writes a job script for the 
job resource manager. The job is submitted to the resource manager and 
the daemon continually checks for the status of the job with "qstat" and 
for messages from AMQP. It should return "live" feedback to the user and 
write to a log.

   [1]: https://twistedmatrix.com/trac/wiki/Documentation
   [2]: http://docs.python.org/2/library/threading.html
   [3]: http://greenlet.readthedocs.org/en/latest/
   [4]: 
https://wiki.openstack.org/wiki/NotificationEventExamples#Immediate_Notifications:
   [5]: http://www.clusterresources.com/torquedocs21/commands/qstat.shtml

Justin Chiu            TRIUMF