[Twisted-Python] Using Twisted with MQ

[Chris Mellon]

On 6/8/07, David Mitchell <monch1962 at gmail.com> wrote:
> Hello all,
>
> I've got what I think is a potential application for Twisted, but I'm
> having trouble figuring out whether Twisted is going to be the best
> path or not.  I've worked through the Twisted Finger tutorial and read
> through the "Ball of snakes" O'Reilly book, but I'm not sure I've got
> my head around Twisted sufficiently.
>
> I have to simulate a server that reads messages off IBM's MQ queues,
> and posts responses back to different MQ queues.  I've used both pymqi
> and Python/COM (on Windows only) to perform this sort of task in the
> past, but now I need something that's going to be able to respond to
> incoming MQ messages at a very high rate.
>
> I suspect I'm going to have to use reactor.callInThread to spawn off
> threads within Twisted in order to handle IO with the MQ queues, as
> I'm not aware of any event-based interface to MQ and therefore any
> reads/writes from queues are potentially going to block for a while.
>
> Is Twisted suited for this sort of application?  Better yet, is anyone
> aware of info I can access regarding this sort of application with
> Twisted? - Google hasn't been much help...
>
> Thanks in advance
>
> Dave Mitchell
>

I haven't written any real code that does this, but I was just
investigating it for feasability myself a couple days ago. I wrote a
thread that looked basically like this:
(pseudocode, as I don't have any of the code in front of me)

while True:
    try:
        res = mq.get() #blocking call, with a fairly long timeout - 5-10 seconds
        reactor.callFromThead(someCallbackFunc, res)
    except Timeout: #pymqi will translate the timeout to an exception
        pass
    if self.exit.is_set(): #threading.Event to check for termination.
        self.terminate()


This just spins happily along, and dispatches processing into the
Twisted event loop. I didn't really test it for performance or
latency. Used reactor.callInThread to dispatch the writes.