[Twisted-Python] Peloton Distributed Service Application Engine

Matthew Pontefract twisted-list at zorinholdings.com
Fri Jul 4 06:09:38 MDT 2008


Hello Twisted Users

I'd like to announce a new project called Peloton that may be of
interest to some of you.

Peloton is a distributed service application platform built in Python on
top of Twisted. It's goals are:

  * To make trivial the process of writing, publishing a method
  * To provide all network interfaces on any relevant protocols and
insulate the service writer from all of this
  * To provide simple, intuitive, modular mechanisms to transform the
results of a method into a form appropriate to the end user
  * To take care of managing services to ensure they stay alive, are
distributed across a grid of computers and have requests distributed
between them
  * To manage failure by re-issuing failed requests and managing the
grid of nodes in such a way that, as far as is reasonably possible, the
client is never troubled to know what lengths were gone to to return a
result
  * To provide scalability
  * To provide inherent parallelisation for trivial problems
  * To provide an event bus accessible to server-side components and clients
  * To provide for easy (hot) upgrade of components and near zero downtime

Why?

I have worked in banks, small firms, big firms, supercomputing firms. I
have worked in Java, in .NET, in Python. In almost all cases I was
writing client-server systems where the application was event driven,
needed to be resilient and would benefit from distribution for
horizontal scalability. In all cases, making the results of server side
operations available to as many different third party components as
possible (via a multitude of protocols) would have been useful. And
nowadays having data easily transformed into snippets of HTML or other
formats, perhaps for AJAX apps etc is increasingly useful.

Peloton is built to leverage the clarity, simplicity, power and rapid
development times of Python and, using the reliable and solid Twisted
core, provide the higher level features that give me all the above in a
small and easy package.

I'm not aiming to re-invent the wheel: I needed an event bus and, having
used many, I've elected to use AMQP providers, RabbitMQ being a
favourite. But this is all switchable because such functionality is
managed by Peloton kernel plugins.

Similar projects to Peloton exist: Pythomnic and Divmod-Mantissa to name
two. They provide some, not all of the functionality or do things in
ways that didn't quite suit me etc.

Peloton is new, but I've implemented the ideas previously with the same
technology at a supercomputing facility, and I presented the work at
PyCon UK 2007. It works, people like it and its use is growing there.
That's good enough for me to know that, at least for a section of the
community, this is probably of interest.

The future

Peloton is very early at the moment and much is to come: a scheduler
(cron for peloton) properly implemented distributed shared-state engine,
 different ways of deploying service code, better management tools, and
more. Tight integration with Java could extend it's use into banking
environments. A suite of services built on Peloton for job scheduling
and management will provide another avenue to explore.

It's all fun, and all interesting. It builds on the truly fantastic
work of others and hopefully makes something novel and interesting. But
that's for you to decide and I invite you to scout around the
documentation, download the source code and give it a whirl. I'd love to
hear what you think, know how you're using it and work with you to make
it better.

Where from?

http://www.peloton-grid.net is the home of Peloton and hosts
documentation and details of how to obtain Peloton. I hope it appeals
and look forward to discussing with some of you about it. We can always
meet on freenode.net, channel #peloton.

Regards
Matthew




More information about the Twisted-Python mailing list