[Twisted-Python] [Q] multi-process chat client and poller
cary.hull at gmail.com
Sat Jan 3 16:14:48 EST 2009
Take a look at ampoule:
On Fri, Jan 2, 2009 at 2:35 PM, V S P <toreason at fastmail.fm> wrote:
> I am in the process of evaluating a tool for the newtorking architecture
> I need. And wanted to ask the group questions, I could not easily get
> answered from reading docs and various posts:
> Basically my architecutre requires the following
> a) multi-CPU/multi-core scalability
> b) multi-machine horizontal scalability
> c) Event dispatching: having a single 'dispatcher' instance
> being able to read rows (events) from postgresql database and
> being able to dispatch it to available 'worker' processes
> d) chat server
> For c) and d) am looking for a framework such that it will allow me
> to develop the 'worker' processes such that they reside on mulitple
> machines, and a lot of the 'hard work' of registration, message passing,
> restarting, SNMP compatibilty .
> Twisted appears to already have message passing, chat server and many
> many other neat things that if I do not need now -- probably will need
> in the future.
> However, I cannot quite understand how the 'multi-process' part is
> I have read this:
> (found it via Bruce Eckel's log)
> and it says that:
> and since forking Python processes has many disadvantages, like Python's
> reference counting not playing well with copy-on-write and problems with
> shared state, it was felt the best option was an event-driven framework.
> I also looked at the docs for the
> But I got confused there
> Basically I am looking to 'pre-spawn' a number of worker threads
> configured from command line (and each of the worker threads will
> precreate a database connection). So each multi-cpu machine will
> have one twisted Server, each server will pre-spawn N worker processes
> and each process will have its own database connection (and therefore
> each can do its own caching and transaction control (or use memchaced
> server) to share cache).
> I did not see however how to:
> a) make the Deferred mechanism to pass the event data received
> in the Asynch loop to one of the worker processes
> (there are appeared to be no 'inheritance' structure to where derive
> the worker processes from).
> b) I did not see how multiple twisted servers, each running on a
> multi-cpu machine can register together to be in one 'cluster' -- so
> my dispatcher process that reads the events from the database can
> 'round-robin' the events to them.
> c) I did not see if twisted 'figures out' that given process runs on a
> local machine vs remote and optimises the IPC communication for local
> Because I did not see the above -- it lead me to believe that I am
> trying to ask Twisted have something it was not meant to do
> (My view of the architecture is somewhat similar to how would ACE/TAO
> ORB notification service would work -- as I am familiar with those).
> I kind of did not go into the multi-box architecutre of how the
> Twisted chart server is working, because I thought I need to understand
> the above first.
> So wanted to ask you guys if I am looking for the right documentation
> or may be there is a separate sub-project that is doing what I am
> looking for.
> thanks in advance,
> V S P
> toreason at fastmail.fm
> http://www.fastmail.fm - Or how I learned to stop worrying and
> love email again
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
01100011 01100001 01110010 01111001
More information about the Twisted-Python