[Twisted-web] leightweight CMS

[Andrea Arcangeli]

On Mon, Mar 14, 2005 at 01:02:58AM -0500, James Y Knight wrote:
> Of course you are correct, but I would disagree about the short-term 
> importance. I suspect you'll get better performance actually by reading 
> files blockingly than by using an SQL server. For a typical website, I 
> expect all the most active content will fit in the cache quite handily, 

A typical website yes, but an application server with potentially a ton
of articles may not allow all data to fit in cache.

And if the cache is flushed by some other task the sync behaviour will
return.

> not matter. If you read in small chunks, the disk will likely be able 
> to read quicker than your kernel TCP buffers can become empty. However, 

Depends on the performance of the HD and of the network. For small/mid
projects it's ok of course, but I doubt people should risk to build
anything significant on top of this model.

> notice my use of the "likely", "expect", "suspect" qualifiers: I have 
> not actually *done* this measurement, but from my experience with SQL 
> servers, they are not excessively speedy.

The object is to allow the high traffic part of the website to be
served while somebody is browsing the old articles that don't fit in
cache.

The latency of the sql server isn't the interesting point here.

> Spawning threads to do your reads is of course an option, but not one I 
> would recommend worrying about unless you've actually measured it being 
> a problem. And, if you *have* done that measurement, please help to 
> write a generic async-disk-io module for Twisted which can use either 
> threads or kernel async IO, as appropriate, and put this to rest once 
> and for all. :)

Sure, I didn't mean people should implement the async-disk-io module
immediatly, but they should be aware that reading or writing within
twisted breaks the async model and that they must be prepared to fix it
if something goes wrong. After the user is aware about this, it's up to
him/her to balance the tradeoff between coding simplicity and the
scalability of his code. It's perfectly ok to go for code simplicity for
small/mid projects that don't get more than 1 connection per second (in
1 second tons of I/O can happen of course).