[Twisted-web] Performance of twisted web with HTTP/1.1 vs. HTTP/1.0

[Andrew Bennetts]

On Mon, Apr 05, 2004 at 12:32:37PM -0500, Jason E. Sibre wrote:
> ACT is part of the Microsoft Visual Studio .NET.  It stands for Application
> Center Test.  The reason I tried it is that it allows the person setting up
> the test to control which headers are sent to the server.  In fact, the test
> is actually controlled programmatically via VBScript (it hooks into Internet
> Explorer to 'record' the test, if you don't want to create it from scratch,
> which would be very tedious).

Ah, ok.  I've played with an older version of it a couple of years ago I
think, back when I was still working for a company that used windows :)  I
think it was called "WAST" (Web Application Stress Tool) or something at the
time, I guess ACT is probably the new version of that :)

Anyway, that explains why I couldn't find it on google :)

> If there's an open source tester that you'd rather I use for this
> discussion, please let me know about it, and I'll chase it down.

Well, any open source tester, even a simple dodgy one you write yourself --
I don't have any strong preference, but it would be nice to be able to
reproduce your results :)

> In the meantime, I'm attaching the test script that is being used
> (test.vbs).  It's written in VBScript, but it can't be run standalone, as
> the "Test" object won't exist unless it's being run from the ACT.  It is
> easy to read however, so you can see what's going on.

Yeah, nothing usual, certainly nothing that looks problematic...

> I've modified a portion of the script that ties Quixote to Twisted, so that
> I can 'peek' at the HTTP headers going into/coming out of Quixote.  Here are
> the headers when things are running fast:
> 
> INBOUND header GATEWAY_INTERFACE: CGI/1.1
[..some headers..]
> INBOUND header SERVER_PROTOCOL: HTTP/1.0
> INBOUND header SERVER_SOFTWARE: TwistedWeb/1.2.0
> 
> 
> And here they are when things are running slow:
> 
> INBOUND header GATEWAY_INTERFACE: CGI/1.1
[..identical headers..]
> INBOUND header SERVER_PROTOCOL: HTTP/1.1
> INBOUND header SERVER_SOFTWARE: TwistedWeb/1.2.0
> 
> I only note two differences, the HTTP/1.0 vs. HTTP/1.1 and the REMOTE_PORT
> 1308 vs. 1445, so, one significant difference.
> 
> Not much help, I guess, but does it give you any clues?

Not really.  I've read through the code in twisted/protocols/http.py, and I
simply don't see any significant differences in the code paths for serving
HTTP/1.0 and HTTP/1.1.

> (I've joined the Twisted-web list for the duration of this conversation, at
> least)

I've CC'd the list.

I can't think of what else to suggest, except for running the server in the
Python profiler (the -p=profile.log switch to twistd, if you're using it),
and seeing if that reveals where the extra time is being spent.

Maybe you could insert a "print repr(line)" into HTTPChannel's lineReceiver
handler (see twisted/protocols/http.py) to get a more raw dump of the
request, and double-check that there's no significant differences in the
HTTP/1.0 vs. HTTP/1.1 requests, but I'm doubtful...

Also, to eliminate another difference between your environment and mine, is
it possible to run your tester against a Twisted server on a linux box, and
verify that it still gives the same behaviour?  If for some bizarre reason
it turns out to be a windows-only issue, it'd be good to know before I waste
too much time trying to reproduce it on Linux :)

Also, I'm losing track of what you have and haven't tried... it might be
time we opened a bug report for this, so we can track this properly.

-Andrew.