[Twisted-Python] [Q] multi-process chat client and poller

[V S P]

Thank you so much
Yes, this plugin is what I am looking for process pool
" 
    """
    This plugin provides ways to create a process pool service in your
    system listening on a given port and interface and answering to a
    given set of commands.
    """
"

So I just need to inherit from 
AMPConnector and have a trully scalable server!


Two more questions:

What's the mechanism of 'registring' multiple twisted servers
(something similar to a Name server in Corba) -- so an event
dispatching client can 'discover' the servers running on 
multiple machines

And how does one run multiple XAMPP servers in twisted -- such
that work to each server is distributed either by a hash on a user
name or some other way (or does such mechanism exist).

thank you again for such a helpful reply.







On Sat, 3 Jan 2009 13:14:48 -0800, "Cary Hull" <cary.hull at gmail.com>
said:
> Take a look at ampoule:
> https://launchpad.net/ampoule
> 
> On Fri, Jan 2, 2009 at 2:35 PM, V S P <toreason at fastmail.fm> wrote:
> > Hi,
> > I am in the process of evaluating a tool for the newtorking architecture
> > I need. And wanted to ask the group questions, I could not easily get
> > answered from reading docs and various posts:
> >
> > Basically my architecutre requires the following
> >
> > a) multi-CPU/multi-core scalability
> >
> > b) multi-machine horizontal scalability
> >
> > c) Event dispatching:  having a single 'dispatcher' instance
> > being able to read rows (events) from postgresql database and
> > being able to dispatch it to available 'worker' processes
> >
> > d) chat server
> >
> >
> > For c) and d) am looking for a framework such that it will allow me
> > to develop the 'worker' processes such that they reside on mulitple
> > machines, and a lot of the 'hard work' of registration, message passing,
> > restarting, SNMP compatibilty .
> >
> > Twisted appears to already have message passing, chat server and many
> > many other neat things that if I do not need now -- probably will need
> > in the future.
> >
> > However, I cannot quite understand how the 'multi-process' part is
> > supported.
> >
> > I have read this:
> > http://www.python.org/workshops/2002-02/papers/09/index.htm
> > (found it via Bruce Eckel's log)
> > and it says that:
> > "...
> > and since forking Python processes has many disadvantages, like Python's
> > reference counting not playing well with copy-on-write and problems with
> > shared state, it was felt the best option was an event-driven framework.
> > "
> >
> > I also looked at the docs for the
> > http://twistedmatrix.com/projects/core/documentation/howto/process.html
> >
> >
> > But I got confused there
> >
> >
> > Basically I am looking to 'pre-spawn' a number of worker threads
> > configured from command line (and each of the worker threads will
> > precreate a database connection).  So each multi-cpu machine will
> > have one twisted Server, each server will pre-spawn N worker processes
> > and each process will have its own database connection (and therefore
> > each can do its own caching and transaction control (or use memchaced
> > server) to share cache).
> >
> >
> > I did not see however how to:
> > a) make the Deferred mechanism to pass the event data received
> > in the Asynch loop to one of the worker processes
> > (there are appeared to be no 'inheritance' structure to where derive
> > the worker processes from).
> >
> >
> > b) I did not see how multiple twisted servers, each running on a
> > separate
> > multi-cpu machine can register together to be in one 'cluster' -- so
> > that
> > my dispatcher process that reads the events from the database can
> > 'round-robin' the events to them.
> >
> > c) I did not see if twisted 'figures out' that given process runs on a
> > local machine vs remote and optimises the IPC communication for local
> > IPC.
> >
> >
> > Because I did not see the above -- it lead me to believe that I am
> > trying to ask Twisted have something it was not meant to do
> > (My view of the architecture is somewhat similar to how would ACE/TAO
> > ORB notification service would work -- as I am familiar with those).
> >
> >
> > I kind of did not go into the multi-box architecutre of how the
> > Twisted chart server is working, because I thought I need to understand
> > the above first.
> >
> > So wanted to ask you guys if I am looking for the right documentation
> > or may be there is a separate sub-project that is doing what I am
> > looking for.
> >
> > thanks in advance,
> > Vlad
> >
> >
> >
> >
> > --
> >  V S P
> >  toreason at fastmail.fm
> >
> > --
> > http://www.fastmail.fm - Or how I learned to stop worrying and
> >                          love email again
> >
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
> 
> 
> 
> -- 
> 01100011 01100001 01110010 01111001
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- 
  V S P
  toreason at fastmail.fm

-- 
http://www.fastmail.fm - mmm... Fastmail...