[Twisted-Python] Twisted enterprise deployment models

Reza Lotun rlotun at gmail.com
Thu Oct 20 07:29:42 EDT 2011


Hi Brad,

On Thu, Oct 20, 2011 at 11:50 AM, Brad Milne
<brad.milne at devx.runthered.com>wrote:

> ...
> So my questions:
> 1) I'm right, right? Twisted doesn't have mechanisms or a library for hot
> swapping code modules or similar solutions for low impact upgrades and high
> availability? (I originally read about Twisted *plugins* hoping for this,
> but was disappointed to find how basic they actually were)
>

No, not a complete, first-class mechanism (at least as far as I know).

However, there are a few pieces in Twisted which can be used to build such a
solution. I've been meaning to piece them together for a while to do this,
but time-constraints and what-not have prohibited that so far:

1.
http://twistedmatrix.com/documents/current/api/twisted.python.rebuild.html
This a module that can be used to reload class definitions and instances,
etc. You can conceive of a system whereby you load a class that defines a
code loading module that can pick up modules in a directory and periodically
refresh their definitions, for example. One way to do it with RabbitMQ is
have a queue specifically for code-refreshes - when a code-reload service
receives a message it does one or more twisted.python.rebuilds to reload the
code definitions, or something. There is obviously still a lot you have to
do yourself, but it's something. (I'm aware that reloading modules in Python
is generally difficult because of all the stray references that can still
exist, but you can probably structure the code in such a way that a reload,
or series of reloads will do the job).

2.
http://twistedmatrix.com/documents/current/api/twisted.runner.procmon.ProcessMonitor.html
A service to monitor processes. You can have other services running in
container processes, and use this functionality to monitor and control them.

3.
http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.ProcessProtocol.html
You can define process protocols that implement specific job types. Usually
this is used to spread process amongst cores, but you could also use it for
sanboxing functionality I suppose. Another project that uses this for
spreading working process amongst cores is Ampoule:
https://launchpad.net/ampoule


2) Is RabbitMQ with txAMQP-empowered Twisted instances on the ends of the
> queues a sensible, compatible, best-practice approach as agreed by the
> community?
>

Sure. It's one approach. You can also use Redis and txRedis -
https://github.com/rlotun/txRedis. Redis has pub-sub functionality that's
much simpler than AMQP (but not as powerful - for example you can't do
something like ACK a task). But for basic pub-sub it's awesome.

We use both RabbitMQ and Redis, via txAMQP and txRedis.

Reza

-- 
Reza Lotun
mobile: +44 (0)7521 310 763
email:  rlotun at gmail.com
work:   rlotun at twitter.com
@rlotun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20111020/6bc4608f/attachment.htm 


More information about the Twisted-Python mailing list