[Twisted-Python] [Q] Naming Service for Twisted Servers

[V S P]

Hi, I thought that 
MQ is quite 'heavy' for my needs 

Basically the whole system allows users to request 'reports'
and those reports can rung for days (or hours).

Instead of having 'one' persistent queue (or a service)
such as provided for typical MQ implementations -- I thought
just to have a 'little' message queue that for each user -- that
would be visible from their profile
(since there is no 'admin' interface to administer 'jobs' as in typical
message queues)


So I just simply planned a process scan each profiles queue,
and send the job requests in round robin to available twisted servers.

You might ask -- well you will store events to be dispatched in 
a table with profile IDs -- so that's what message queues do anyways.

However, the system is built such there is more than one database
instance
(where users are destributed based on a hashing scheme acrsoss multiple
databases) -- so it is not simply one table (that's also why there is
a need for multiple Twsited servers  as they would serve their own
'partitions' in the system)


On the other hand ... may be I need to look at open source and free
MQ solutions to see if I is easier to 


Thank you again for all the suggestions so far,
I am writting up a small summary of the replies for
furthere analysis.

On Sat, 10 Jan 2009 14:27:18 -0800, "Don Smith" <donwsmith at gmail.com>
said:
> Why not use a Message Queue? I developed a very similar program to what
> you
> have described, only each agent would connect to the Queue to get their
> data
> to work on. However, I think something like Apache ActiveMQ or AmazonSQS
> would be a better approach.
> 
> =Don
> 
> 2009/1/9 V S P <toreason at fastmail.fm>
> 
> > Thank you for the replies.  Wanted to make sure I was not missing
> > a 'twisted goodie'
> >
> > What is ZeroConf?  Can you please point me to the project page
> > (I searched for some but did not find something twisted/python
> > specific).
> >
> > my client will be a program that reads rows of 'requests' from the
> > database and dispatch them to available twisted servers (I call it
> > Dispatcher).  Then twisted servers will use the pool of worker threads
> > (with help of Ampula plugin I guess) to do the work (that will take
> > hours
> > of not more).
> >
> > So there will be just one program that needs to be
> > aware of all the available servers in my scenario.
> >
> > Also I can see that some people might have needed a 'proxy'
> > that sits infront of the cluster of twisted servers and that proxy
> > has some balancing mechanism.  And if there is any body that has that
> > solution I would like to know as well.
> >
> >
> >
> > thank you again.
> >
> >
> > On Fri, 9 Jan 2009 22:29:22 -0500, "Jarrod Roberson"
> > <jarrod at vertigrated.com> said:
> > > 2009/1/9 V S P <toreason at fastmail.fm>
> > >
> > > > Hi,
> > > > I am sorry I am reasking the same question I asked before
> > > > (but I did not get any replies)
> > > >
> > > > Basically wanted to know -- for running Twisted on multiple servers
> > > >
> > > > is there such a thing as a "Naming" server that
> > > > would register all the currently running twisted servers
> > > >
> > > > -- so that asynchronious event dispatching can be done in some
> > > > round-robin to available twisted servers
> > > >
> > > >
> > > > we used ZeroConf for autodiscovery that way the clients can decide for
> > > themselves which server to connect to.
> > --
> >   V S P
> >  toreason at fastmail.fm
> >
> > --
> > http://www.fastmail.fm - And now for something completely different┘
> >
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
-- 
  V S P
  toreason at fastmail.fm

-- 
http://www.fastmail.fm - IMAP accessible web-mail