[Twisted-Python] [Q] multi-process chat client and poller

[V S P]

Hi, 
I am in the process of evaluating a tool for the newtorking architecture
I need. And wanted to ask the group questions, I could not easily get
answered from reading docs and various posts:

Basically my architecutre requires the following

a) multi-CPU/multi-core scalability

b) multi-machine horizontal scalability

c) Event dispatching:  having a single 'dispatcher' instance
being able to read rows (events) from postgresql database and 
being able to dispatch it to available 'worker' processes

d) chat server


For c) and d) am looking for a framework such that it will allow me
to develop the 'worker' processes such that they reside on mulitple
machines, and a lot of the 'hard work' of registration, message passing,
restarting, SNMP compatibilty .

Twisted appears to already have message passing, chat server and many
many other neat things that if I do not need now -- probably will need
in the future.

However, I cannot quite understand how the 'multi-process' part is
supported.

I have read this: 
http://www.python.org/workshops/2002-02/papers/09/index.htm
(found it via Bruce Eckel's log)
and it says that:
"... 
and since forking Python processes has many disadvantages, like Python's
reference counting not playing well with copy-on-write and problems with
shared state, it was felt the best option was an event-driven framework.
"

I also looked at the docs for the 
http://twistedmatrix.com/projects/core/documentation/howto/process.html


But I got confused there


Basically I am looking to 'pre-spawn' a number of worker threads
configured from command line (and each of the worker threads will
precreate a database connection).  So each multi-cpu machine will
have one twisted Server, each server will pre-spawn N worker processes
and each process will have its own database connection (and therefore
each can do its own caching and transaction control (or use memchaced
server) to share cache).


I did not see however how to:
a) make the Deferred mechanism to pass the event data received
in the Asynch loop to one of the worker processes
(there are appeared to be no 'inheritance' structure to where derive
the worker processes from).


b) I did not see how multiple twisted servers, each running on a
separate
multi-cpu machine can register together to be in one 'cluster' -- so
that
my dispatcher process that reads the events from the database can
'round-robin' the events to them.

c) I did not see if twisted 'figures out' that given process runs on a
local machine vs remote and optimises the IPC communication for local
IPC.


Because I did not see the above -- it lead me to believe that I am
trying to ask Twisted have something it was not meant to do 
(My view of the architecture is somewhat similar to how would ACE/TAO
ORB notification service would work -- as I am familiar with those).


I kind of did not go into the multi-box architecutre of how the 
Twisted chart server is working, because I thought I need to understand
the above first.

So wanted to ask you guys if I am looking for the right documentation
or may be there is a separate sub-project that is doing what I am
looking for.

thanks in advance,
Vlad




-- 
  V S P
  toreason at fastmail.fm

-- 
http://www.fastmail.fm - Or how I learned to stop worrying and
                          love email again