[Twisted-Python] twisted, SMP and clusters

[Doug Farrell]

Thanks for the response. I was thinking Pyro and PP might be
incompatible with Twisted, since they already have communication
mechanisms in them, but wasn't sure. I've also got a response from Ed
that he might try to get some sample code up on the AsynCluster site to
show how something like this could be implemented. I've got to better
define my goals and design and perhaps post that.

Again, thanks for the response,
Doug

-----Original Message-----
From: twisted-python-bounces at twistedmatrix.com
[mailto:twisted-python-bounces at twistedmatrix.com] On Behalf Of
glyph at divmod.com
Sent: Monday, December 17, 2007 5:17 PM
To: Twisted general discussion
Subject: Re: [Twisted-Python] twisted, SMP and clusters

On 07:44 pm, dfarrell at mypublisher.com wrote:
>I'm considering using Twisted to build a soft realtime system to 
>control a workflow for a physical production facility. Although there 
>would be some socket work for it to do (XML-RPC, logging, etc) I also 
>like it because of the nice way it can be daemonized. One of my goals 
>for this project is to take better advantage of our multiple Linux 
>servers, each of which has multiple CPU's with dual cores. I'd like to 
>take our workflow objects that are fairly long running state machines 
>and cut them loose on other cores/CPUs/servers if I can. So here's my 
>question: should I be looking at twisted.spread for this kind of work? 
>Can the Parallel Python or Pyro projects be incorporated into Twisted, 
>and if so, what kinds of things should I watch out for, or is it even 
>possible? I'm thinking if I use Parallel Python or Pyro I could make 
>remote calls to our objects in a DeferToThread object and get back the 
>results/errors via callbacks. I'm still really pretty new to Twisted, 
>so forgive my nave questions if they are so.

PB (i.e. twisted.spread.pb) might be appropriate to this sort of work. 
It makes communicating between objects in multiple Twisted processes 
fairly easy.

Depending on your requirements, so might a lot of Twisted things.  If 
other code is talking to this, AMP (twisted.protocols.amp) or XMLRPC 
(twisted.web.xmlrpc) might also be appropriate.

Pyro and PP, while potentially valid approaches to your problem, are 
more or less incompatible with Twisted.  They *could* be made to work 
togther, there's no actual technical incompatibility, but they're geared

towards working with entirely different programming paradigms.

When it's all said and done, none of these things is really a turn-key 
distributed programming environment.  They're just different ways to 
communicate messages between nodes.  In order to say what would be most 
appropriate to your problem domain, you'd need to describe your 
environment and your requirements a bit better.  For obvious reasons, 
though, you're not likely to get a suggestion other than "use Twisted" 
in this particular venue :).

It sounds like Ed's AsynCluster might provide the sort of structure (on 
top of PB) that you're looking for, but I don't know much about it, and 
I've never used it.  However, unlike these other tools, it's not just a 
messaging system, it is specifically for doing cluster computing.

_______________________________________________
Twisted-Python mailing list
Twisted-Python at twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python