[Twisted-Python] [Q] multi-process chat client and poller

V S P toreason at fastmail.fm
Tue Jan 13 23:56:22 MST 2009


Hi,
thank you for the reply, yes I meant XMPP and thought that it might
be written to be deployed on multiple machines.  But from what you
are saying it is not.

Yes, started looking at Erlang as well (and its Mnesia component)
rabbitMQ seems like what I need in terms of scalable multi-machine
support.

Basically what I need is to request reports, run reports (for long time,
joining database and file data(where files can be in netCDF4)),
 and then upload results (some times in excel, some times other
formats) back to user-specific database.

I certainly do not want to develop 'middleware' -- I just wanted
to use it.  

And Twisted/Python had bunch of libraries (including excel support,
math libraries, NetCDF4, pyTables (that I will need) ), books, community
support, etc -- compared to Erlang.

So I just was missing a multi-machine distribution scheme... I thought
the Twisted community would run into something similar and have
solutions (google-apps have to be using something to as well to run
on multiple machines)

Another option for me of course TAO ORB, or as you pointed out Erlang
RabbitMQ (especially since Erlang was designed to do this), or as people
suggested one of the open source Message Queuing systems.

In the selection process, I just wanted to stay within Python (rather
than C++, Java, Erlang or a mix of those).

Another option I also thought about (if Twisted would be too
difficult to use for multimachine support) would be to use
Tao ORB, and for each Corba Server simply use Boost C++ Python
binder and then pretty much have all the python stuff I need.

However, there are unknowns there too ... while I am pretty
comfortable with C++, I do not know how Python Database access libraries
will behave when invoked from with TAO's Corba servers...
is there connection pooling there/etc.  Also may be there gottchas
there I do not even think of now.

Anyways, this is not twisted specific... just wanted to explain
why I was looking.








On Tue, 13 Jan 2009 20:42:24 -0500, "Drew Smathers"
<drew.smathers at gmail.com> said:
> On Tue, Jan 13, 2009 at 4:11 PM, V S P <toreason at fastmail.fm> wrote
> > Hi, thank you for the correction on how to use Ampula.
> >
> > Naming service serves
> > really as a 'discovery' tool -- and then the participating
> > components talk to each other.  When new component comes
> > on online, it registers, and the Naming Service broadcasts
> > the new updates to all the subscribed components.  It also
> > can 'ping' the components to see if they are alive and if not
> > it deletes an entry and broadcasts the updates to the remaning
> > components.
> 
> It's sounds like what you want is erlang.  I'm only saying this
> half-seriously.  Neither python or twisted have builtin mechanisms for
> doing what you want.
> 
> > This way the message dispatchers (or components themselves) have
> > almost up-to-date information about their peers on the network (the  is
> > a delay controlled by ping interval).
> 
> Hmmm ... message dispatchers.  Are you building messaging middleware?
> If so, you should probably stop now.  Here's something that already
> works: http://www.rabbitmq.com
> 
> >
> > The naming service data is sometimes stored on a file system
> > (in case it crashes and needs to restart -- sort of like a
> > 'session') with Python's serialization it has to be easy to do.
> >
> > But yes, a naming service while not a bottleneck --  can be
> > a single point of failure in a distributed env, and that's why
> > for 'fault safety' it is typically ran on fail-over hardware
> > (although since it is not doing much, it rarely crashes :-))
> >
> > I certainly agree that using a naming service as a mechanism
> > to get a 'context' of a remote process to invoke its methods
> > (sort of get a Prospective Broker object from a naming service)
> > every time a call is made -- would be inefficient and not correct.
> >
> > But if each component has the table of the registered components,
> > then simply using that table to insanciate a Prospective Broker's
> > RemoteReference for the needed remote component -- would be nice.
> >
> >
> > Few people pointed out that a more generic service discovery
> > systems exist and they use Dbus protocol (they are used to discover
> > printers on the network, fileshares, etc) -- I have to study those.
> >
> > As far as XAMPP, I meant to ask: in twisted, does it run distributed
> > on multiple machines (that is multiple xampp servers each on its own
> > machine) -- if yes -- I thought it would have to have some kind of a
> > naming service built in (I did not mean that XA (AMP) in the naming
> > suggested that it is derived from Ampula :-)).  But may be it is
> > single-machine/single-thread process only (I just did not know).
> >
> 
> Still, what is XAMPP?  I think Valentino was assuming you mispelled
> XMPP.  To partially answer this question though (excluding XAMPP as a
> component), Twisted has protocols (AMP, PB, etc.) that make relatively
> easy to build simple distributed systems (running on multiple
> machines).  If you study hard and work hard you can also build more
> complex distributed systems with grander scalability requirements.
> Twisted does not magically cluster itself on multiple machines.
> 
> -Drew
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- 
  V S P
  toreason at fastmail.fm

-- 
http://www.fastmail.fm - Choose from over 50 domains or use your own





More information about the Twisted-Python mailing list