[Twisted-Python] twisted, SMP and clusters

[glyph at divmod.com]

On 07:44 pm, dfarrell at mypublisher.com wrote:
>I'm considering using Twisted to build a soft realtime system to 
>control a workflow for a physical production facility. Although there 
>would be some socket work for it to do (XML-RPC, logging, etc) I also 
>like it because of the nice way it can be daemonized. One of my goals 
>for this project is to take better advantage of our multiple Linux 
>servers, each of which has multiple CPU's with dual cores. I'd like to 
>take our workflow objects that are fairly long running state machines 
>and cut them loose on other cores/CPUs/servers if I can. So here's my 
>question: should I be looking at twisted.spread for this kind of work? 
>Can the Parallel Python or Pyro projects be incorporated into Twisted, 
>and if so, what kinds of things should I watch out for, or is it even 
>possible? I'm thinking if I use Parallel Python or Pyro I could make 
>remote calls to our objects in a DeferToThread object and get back the 
>results/errors via callbacks. I'm still really pretty new to Twisted, 
>so forgive my naïve questions if they are so.

PB (i.e. twisted.spread.pb) might be appropriate to this sort of work. 
It makes communicating between objects in multiple Twisted processes 
fairly easy.

Depending on your requirements, so might a lot of Twisted things.  If 
other code is talking to this, AMP (twisted.protocols.amp) or XMLRPC 
(twisted.web.xmlrpc) might also be appropriate.

Pyro and PP, while potentially valid approaches to your problem, are 
more or less incompatible with Twisted.  They *could* be made to work 
togther, there's no actual technical incompatibility, but they're geared 
towards working with entirely different programming paradigms.

When it's all said and done, none of these things is really a turn-key 
distributed programming environment.  They're just different ways to 
communicate messages between nodes.  In order to say what would be most 
appropriate to your problem domain, you'd need to describe your 
environment and your requirements a bit better.  For obvious reasons, 
though, you're not likely to get a suggestion other than "use Twisted" 
in this particular venue :).

It sounds like Ed's AsynCluster might provide the sort of structure (on 
top of PB) that you're looking for, but I don't know much about it, and 
I've never used it.  However, unlike these other tools, it's not just a 
messaging system, it is specifically for doing cluster computing.