[Twisted-Python] [Q] multi-process chat client and poller

[Cary Hull]

Take a look at ampoule:
https://launchpad.net/ampoule

On Fri, Jan 2, 2009 at 2:35 PM, V S P <toreason at fastmail.fm> wrote:
> Hi,
> I am in the process of evaluating a tool for the newtorking architecture
> I need. And wanted to ask the group questions, I could not easily get
> answered from reading docs and various posts:
>
> Basically my architecutre requires the following
>
> a) multi-CPU/multi-core scalability
>
> b) multi-machine horizontal scalability
>
> c) Event dispatching:  having a single 'dispatcher' instance
> being able to read rows (events) from postgresql database and
> being able to dispatch it to available 'worker' processes
>
> d) chat server
>
>
> For c) and d) am looking for a framework such that it will allow me
> to develop the 'worker' processes such that they reside on mulitple
> machines, and a lot of the 'hard work' of registration, message passing,
> restarting, SNMP compatibilty .
>
> Twisted appears to already have message passing, chat server and many
> many other neat things that if I do not need now -- probably will need
> in the future.
>
> However, I cannot quite understand how the 'multi-process' part is
> supported.
>
> I have read this:
> http://www.python.org/workshops/2002-02/papers/09/index.htm
> (found it via Bruce Eckel's log)
> and it says that:
> "...
> and since forking Python processes has many disadvantages, like Python's
> reference counting not playing well with copy-on-write and problems with
> shared state, it was felt the best option was an event-driven framework.
> "
>
> I also looked at the docs for the
> http://twistedmatrix.com/projects/core/documentation/howto/process.html
>
>
> But I got confused there
>
>
> Basically I am looking to 'pre-spawn' a number of worker threads
> configured from command line (and each of the worker threads will
> precreate a database connection).  So each multi-cpu machine will
> have one twisted Server, each server will pre-spawn N worker processes
> and each process will have its own database connection (and therefore
> each can do its own caching and transaction control (or use memchaced
> server) to share cache).
>
>
> I did not see however how to:
> a) make the Deferred mechanism to pass the event data received
> in the Asynch loop to one of the worker processes
> (there are appeared to be no 'inheritance' structure to where derive
> the worker processes from).
>
>
> b) I did not see how multiple twisted servers, each running on a
> separate
> multi-cpu machine can register together to be in one 'cluster' -- so
> that
> my dispatcher process that reads the events from the database can
> 'round-robin' the events to them.
>
> c) I did not see if twisted 'figures out' that given process runs on a
> local machine vs remote and optimises the IPC communication for local
> IPC.
>
>
> Because I did not see the above -- it lead me to believe that I am
> trying to ask Twisted have something it was not meant to do
> (My view of the architecture is somewhat similar to how would ACE/TAO
> ORB notification service would work -- as I am familiar with those).
>
>
> I kind of did not go into the multi-box architecutre of how the
> Twisted chart server is working, because I thought I need to understand
> the above first.
>
> So wanted to ask you guys if I am looking for the right documentation
> or may be there is a separate sub-project that is doing what I am
> looking for.
>
> thanks in advance,
> Vlad
>
>
>
>
> --
>  V S P
>  toreason at fastmail.fm
>
> --
> http://www.fastmail.fm - Or how I learned to stop worrying and
>                          love email again
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
01100011 01100001 01110010 01111001