[Twisted-web] Nginx vs Twisted Web

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Tue Jul 2 05:26:22 MDT 2013


On 03:08 am, lozinski at freerecruiting.com wrote:
>I am just choosing a web server, and I thought it would be good to
>consult with this mailing list, since twisted web was on my short list.
>
>I run a python moinmoin wiki, a static site, a zope 2 site, and a zope
>3 site on a VPS.
>
>The thing I care about most, is ease of use.  None of my websites are
>very high traffic.  And I have way way too much complexity in my life 
>to
>manage.
>
>The next thing I care about is reliability.

Twisted Web is rather reliable.  http://twistedmatrix.com/ has run on 
Twisted Web for many years now.  From time to time an issue arises, but 
my experience is that it requires less attention than an Apache 
deployment.  This is just my experience, of course.
>The thing I care least about is performance.  Would that I had that
>problem.

That is an excellent attitude!  I'll just note that Twisted Web is fast 
enough for just about anything you might want to do, though.  More on 
this later.
>So Apache is the obvious first choice.  But it is so complicated.  They
>threw in everything but the kitchen sink.  I just do not like it, nor 
>do
>I want to spend the time wrestling with it to get it to do the needful.
>And you have to restart apache everytime the config files change.

You'll almost certainly have to restart your Twisted Web server every 
time you want to change its configuration.  Nothing stops you from 
building a server using Twisted Web for which this is not the case, but 
it's a lot of careful, detail-oriented work for not a lot of gain.
>[snip - lighttpd etc]
>
>TwistedWeb is very interesting.  When I read the Apache documentation,
>my eyes glaze over.  When I read the twisted documentation, my eyes
>light up.
>What an intelligent object model of networking.  Twisted Web is also
>closely related to Zope.  At the very least, it is also using Zope
>Component Architecture, which I like.

Minor correction, Twisted Web uses Zope Interface (as does the rest of 
Twisted) and some of the same ideas as ZCA, it doesn't actually use ZCA 
itself.
>So here is my comparison of twisted web and nginx.
>
>With Twisted web, first I wonder about configuration.  Nginx is just
>some configuration files.  Perhaps easier then editing a python 
>program.

As I think you've learned, Twisted Web doesn't come with a set of 
standard configuration files.  It is configurable using Python.  It's up 
to you to build a maintainable configuration system out of this base. 
http://twistedmatrix.com/ uses something very descriptively named 
"twisted-web" for configuration.  This is a vaguely (*very* vaguely) 
apache-style configuration system, but it's still built out of Python 
and it's also not currently maintained as far as I know.

Fortunately configuration of most features of Twisted Web only takes a 
couple lines of Python at most.  I find this doesn't get too bad.  Also, 
I tend to follow the rule that when configuration does get longer than a 
few lines, it is time to turn it into *software* with documentation and 
unit tests, and then *use* it from the configuration.
>Then I wonder about reliability.  Nginx has a larger installed base,
>more bug reports presumably more reliable.   Plus it includes a process
>monitor that
>keeps things running.  I think twisted web includes monitoring 
>software,
>but perhaps not as robust and mature.

Twisted Web doesn't include any monitoring software, although Twisted 
Runner does include "procmon" which launches and re-launches a program 
as necessary, so perhaps this is what you're referring to here.

There are also lots of other projects that do what "procmon" does (as 
well or better) and are entirely compatible with Twisted Web, so there's 
no shortage of options here.

Returning to http://twistedmatrix.com/, the Twisted Web server deployed 
there does *not* run as a managed process controlled by one of these 
types of tools.  It just keeps running and never needs restarting.

What kind of reliability concerns do you have apart from the process 
aborting, segfaulting, or otherwise crashing due to implementation bugs?
>The next nice thing about Nginx is that you can drop configuration 
>files
>and the server recognizes them, and does the right thing.  No need to
>reboot.
>The twisted web python program I would have to restart.

Yep.
>Finally I worry about the performance.  You can see a bunch of
>interesting results for python servers here.
>http://nichol.as/benchmark-of-python-web-servers
>
>In general twisted web does not seem that fast, nor reliable.  High
>error rates.  Bummer.

Please disregard those results. :(

First, all of the servers he benchmarked performed remarkably well.  500 
requests/second translates to 2ms/request of overhead.  Perhaps your 
moinmoin requests will take 5ms and your zope requests will take 15ms (I 
just made those number up, I have no idea how moinmoin or zope perform). 
So your per-request cost is now 7ms and 17ms - that's 142 
requests/second and 58 requests/second respectively.  I suspect that's 
fast enough for you, but the real point is to compare this to an ideal 
server with 0ms/request of overhead.  On such a server, you would get 
200 requests/second on moinmoin and 66 requests/second on zope.  Is the 
difference between 142 and 200, or 58 and 66 significant for you?

Second, no attempt has been made to explore or explain what "errors" 
mean in the survey.  That said, my suspicion is that they're caused by 
high load on the platform TCP/IP stack and would not typically be 
observed in real world usage where clients *don't* run on the same host 
as the server.  Regardless, as there's no discussion on the write-up and 
in real-world use I haven't observed problems like this, I have assumed 
the errors are spurious and due to the test setup (and await a 
correction from anyone who can demonstrate otherwise).

Third, the benchmark use the WSGI system of Twisted Web.  Perhaps WSGI 
is the only feature you care about in a web server (if so, then you 
don't need to think too hard about picking a WSGI container, since they 
are extremely interchangeable - if the one you pick has a problem, 
switch to another, it's the work of 15 minutes or so at worst). 
However, no attempt has been made to configure the system for maximum 
performance (the default threadpool size is used - a value tuned for DNS 
requests, not WSGI threads) and the benchmark comparison puts this up 
against several "fake" WSGI implementations ("lightweight threads" and 
"callback/generator" flavors are not real WSGI containers).  If you 
wanted to push performance higher, there's every reason to expect you 
could.  And if you don't use WSGI, then performance automatically jumps 
dramatically.
>What is my conclusion?  Well I think Twisted web is a brilliant object
>model for networking software.  But dedicated software for web serving,
>like nginx performs way better
>and is much more reliable.

If you go into more detail about what you mean by "reliable" here, 
someone may be able to speak to it.  As I mentioned above, the 
deployments of Twisted Web I've managed are quite reliable, they just 
keep working year after year with minimal maintenance.
>So I think I am going to use Nginx in front of my moinmoin, zope 2, 
>zope
>3 and static servers.   Is that the right decision?   Pity, I really
>like twisted web.
>
>Anyhow I thought I would post this here, and see what people have to
>say.  Presumably I made some mistakes in this document.  Please let me
>know what they are.

Thanks for writing this up and sending it to the list.  I hope some of 
my responses have been useful or at least interesting.  Good luck in 
whatever you choose to deploy!

Jean-Paul



More information about the Twisted-web mailing list