[Twisted-web] /freeform_post!!random causes exceptions

Andrea Arcangeli andrea at cpushare.com
Tue Jan 11 03:27:51 MST 2005


On Mon, Jan 10, 2005 at 01:14:31PM -0800, Donovan Preston wrote:
> box "Application server" which encompasses web serving, session 
> management and persistence, are slated for inclusion in a new 
> "application server built on top of twisted.web, nevow, and atop" 

The way you're modelling the webserver with atop is not correct.  You
can't do synchronous I/O in twisted context.  That's fine for the
examples directory to keep them simple, but if you're planning something
more serious than the example directory, you just can't go that way (I
mean with a database that will not be loaded all in ram when you start
the application). As a workaround you should at least use: find -type f
-exec cp {} /dev/null \;, so that if you've enough cache it may help,
but that cache will be polluted during backup, so it's not going to work
well as a workaround.

To provide a scalable framework you must create a separate thread and
pass up the Item with perspective broker. I considered using bsddb and
pass up the item with a remote object with PB myself when writing my
server, but at the end it has been an order of magnitude a better
decision to give it up and switch to sql with psycopg2 (suggested by
Valentino). Of course I don't get automatic persistence, but atop also
doesn't provide it, and if I've to call touch, I can as well call
.flush() and .flush() will run the sql code to give the object
persistence.

In theory the ideal for you would be to support the aio support in the
kernel, that gives you everything you need, but that will require a
significant change in twisted, since twisted will have to use sleep in
aio_complete and not in poll anymore (and all poll calls will have to be
submitted with io_submit).

Then there's a major conflict between epoll and waiting for aio
completion, so it's not like the above would be a final solution, and I
give epoll more priority on the twisted side which is the webserver.
Plus having the db in a separate task will scale better in smp (nevow is
quite cpu hungry).

The bottom line is: at the moment using aio would be an huge
complication and it's worthless to do it until we can use epoll and aio
at the same time. So if you want to use atop for something big, you've
to put it in a different task, and that will make it a lot more
complicated than a normal sql solution (at least this was the case for
me).

Even the session persistence I'm going to provide it with psycopg2 (not
atop) for the same reason. Plus eventually as Valentino suggested me I
will be able to have multiple webservers sharing the same db, and the
user won't notice switching from one to the other since they've zero
persistence (all persitence is provided by psycopg2, including the
session cookie storage).

Actually my proprietary app involves more servers than just the
webserver and db, the above is only a short description of the web side
of it.



More information about the Twisted-web mailing list