[Twisted-Python] Twisted protocol as Django data source

[glyph at divmod.com]

On 07:40 pm, exarkun at divmod.com wrote:
>On Wed, 27 May 2009 14:39:45 -0400, Dan <charnedn at muohio.edu> wrote:
>>Hi all,
>>
>>I have an application written in Twisted which multiplexes data from a
>>variety of pieces of specialized hardware, and I'd like to make the
>>live stream of this information viewable through a webpage.
>>Unfortunately, on our server, port 80 is already taken by an Apache +
>>mod_python + Django stack with which we host a number of webpages.
>>
>>[snip]
>>
>>The options I've thought of include using Twisted's builtin web server
>>a nonstandard port, mod_proxy, or writing a Django app which just
>>polls a database repeatedly and altering my Twisted app so it
>>repeatedly does an UPDATE. All of these solutions feel hacky to me, so
>>I'd like to ask you, the list, how you'd go about doing this.

Polling a database repeatedly and updating it "repeatedly" to 
synchronize data sounds pretty gross.  But then, if your Django 
application is rendering static web-pages and not updating them with 
some COMET technique, you can simply populate the database from Twisted 
when stuff changes and query it normally when the page is rendered - 
that doesn't seem bad to me.

If your Django application *is* using COMET to keep the page updated in 
real time, then you should look into Orbited - which, as it so happens, 
uses Twisted, so you can put it into your Twisted process.

Personally, I don't like using databases as a point of integration. 
Inevitably your Django app or your Twisted app will want to enforce 
constraints on the data and model things about the relationships between 
rows beyond what one can glean by inspecting the SQL schema.  Once the 
responsibility for managing the data's integrity has moved into 
middleware, you should be talking to the middleware and let that 
middleware manage your database.  Given that your Twisted process is the 
thing doing the UPDATEs here, I'd treat Twisted as the middleware, and 
agree with exarkun's recommendation:
>Another option is to have Django talk to a Twisted process via some RPC
>mechanism that won't require you to use Twisted in the Apache process.
>For example, XML-RPC.

In this vein, there is also a blocking-friendly implementation of the 
AMP protocol with no Twisted dependency:  https://launchpad.net/ampy - 
but it's maintained outside of Twisted and I wouldn't vouch for its 
quality.  On the other hand I wouldn't vouch for the quality of any XML- 
RPC client either ;-).

You can use whatever protocol you want for exchanging information 
between these two systems.  The important thing is really to make it a 
narrow, clearly-specified channel of communication.