[Twisted-web] Nevow future

[glyph at divmod.com]

On Tue, 11 Apr 2006 17:56:18 -0200, Manlio Perillo <manlio_perillo at libero.it> wrote:
>Jean-Paul Calderone ha scritto:

>- what about formless?

Pollenation's "Formal" seems to be replacing it as a de-facto standard in the community for people who want form generation.  It may even be distributed with Nevow at some point.  However, formless itself is somewhat of a dead end; while it contains some handy convenience utilities to slap a form on a page, you end up having to do almost as much work to get a decent-looking form with formless as just writing the HTML by hand yourself, and the rules are often implicit and undocumented.

>> When there is a 1.x release, you can begin to expect things
>> to stop changing as much.

>There are previsions?

What is a 'prevision'?

>>> It will remain indipendent from the underlying web server?
>>>
>>
>> It isn't independent from the underlying web server now, so saying that
>> it would remain that way doesn't make sense.

>Well, there is support for WSGI and an implementation with Zomne...

The WSGI support may not work at all.  It is unsupported, undocumented, and there are no tests which import it.

The WSGI support will only realistically work in a Twisted-style environment.  Athena depends explicitly on the reactor, so invoking any Athena-using applications in a non-Twisted context will not work.  Livetest requires the reactor as well, so you will not be able to write nit tests for your application or use any 3rd party fragments which use athena or are tested in this manner.

It might be _possible_ to write a Nevow application through wsgi, but why bother?  Unless you already know Nevow intimately and have some bizarre deployment constraints, it is unlikely that it is the best tool for that job.

"zomne" is a way to run your twisted.web server in "stealth mode" behind another webserver.  It's also untested, and depends on a C component, which I believe the author did as a simple proof of concept to answer the question "but my $5 hosting doesn't let me run a long-running server", not a robust deployment mechanism.

Don't get me wrong, I would love it if there were dedicated maintainers for these use-cases, since I realize they are important to many people.  They just aren't important to *us*, so we're not going to spend time.

>> Your Nevow applications should remain independent of it, though.

>I would like to use lighttp as the *only* webserver, and Nevow *only*
>for the rendering, session support, and so.

lighttpd supports reverse proxies.  Nevow's session support, in nevow.guard, depends on the reactor and twisted.web, so you will be unable to use it otherwise.

>>> Reverse Proxy is not an option for me (*why* to use two or more
>>> webservers and duplicate HTTP protocol handling?).

>> Because it solves a problem?  Why *not* do this?

>I don't like the idea.

Yes but _WHY_ don't you like the idea?  Do you live in some alternate reality where the arrangement of bytes in an HTTP packet has harmful effects but the arrangement of bytes in a FastCGI packet does not?

>Is this really necessary?

No, you could do away with LightTPD and then you wouldn't need a proxy ;-).  (Seriously though LightTPD is unbelievably fast for static content so dealing with the proxy is probably a good idea.)

>>> Why there is not support for FastCGI (so I can use lighttp)?
>>
>> Because no one has implemented it.
>
>Well, why?

We don't need it.  We use twisted web, or reverse proxies, which solve all of the problems you have raised neatly.

>Is FastCGI really hated or flawed?
>The author of twisted.web2.channel.fastcgi seems not to like it.

It is highly complex, and serves no purpose.  What is the goal of FastCGI?  To communicate between a web server and web applications.  I.E. to relay information about HTTP requests between a program which parses HTTP requests, and a program which acts upon the information present in an HTTP request.  Sometimes -- efficiency and security being the primary concerns here -- different programs have to handle different HTTP requests depending on their content, so a front-end server is required.

One way to do this is to reverse-proxy: I.E. take the HTTP request from the browser, identify the program which will handle it, and hand it off to that program in as close to the original format as possible, in standard HTTP, for minimal loss of information.

Another way to do this is the (Fast)CGI model: take the HTTP request, convert it into a random, half-documented sequence of bytes and/or and environment variables, and then send it to a program whose job is to process HTTP requests but, for some reason, cannot contain a parser for HTTP requests but *can* contain a parser for random mashed-up remnants of an HTTP requests over a different protocol.

This is a common mistake, because it seems "simpler" to design your own protocol for talking about HTTP than to use HTTP itself.  It isn't.  In fact, by definition, it cannot be: your new protocol has to provide all the information present in the original HTTP request (or risk breaking applications that depend on some information you did not relay), so in the best case it can be exactly as complex as HTTP.  I wrote something like this in twisted.web.distrib, where I converted the HTTP requests to PB and back again.  While the "automated proxying for users" functionality was cool, I had *just written* an HTTP parser and it was silly not to just use it over a unix socket.  I am pleased to note that web2, when it implements something like web.distrib, will do it the correct way and just make it a regular reverse proxy over a unix socket.

FastCGI itself is also actually harder to parse, contains numerous design errors, and more confusing and inconsistent than HTTP itself.  There are numerous inconsistencies it could have fixed in HTTP - it could have properly optimized the multiplexed-channel case with inband flow control to avoid TCP startup overhead for distant FastCGI proxies - it could have normalized the transport over something easy to audit and implement like netstrings - but it totally failed to do anything worthwhile despite being an entirely new protocol.

By contrast, SCGI has the same property of "transmitter of HTTP data that isn't HTTP" but is not insanely overcomplex and full of mistakes.  It at least works enough like CGI that it caters to a market of people writing server containers who can't necessarily write a good HTTP implementation -- whereas if you have enough time to implement FastCGI, you can _definitely_ implement HTTP.

Nevertheless, dumb as it is, supporting FastCGI is still worthwhile.  Various bits of useful software have integrated it for one reason or another, and some of them don't have other ways of talking to the outside world.  That doesn't mean it's a good idea to write new software that is similarly hamstrung by a bad integration decision.

>> Feel free to contribute a FastCGI
>> implementation, if you like.  Efforts might better be spent finishing
>> twisted.web2 (this includes writing documentation, test cases, examples,
>> etc, not just coding) or figuring out how best to port Nevow to web2,
>> since web2 supports FastCGI already.

I would go so far as to say that web2 is _the_ way we will support FastCGI.  It might be worthwhile to support, but we need to move Nevow to web2 at some point, and any effort directed at this should be spent improving web2.

>Now the twisted.web2 implementation support only the Responder role,
>what about the Authorizer and Filter roles?

You can contribute implementations if you like, but does anyone really use those?

>I'm very interested in this but I'm not an expert.
>In the end, is FastCGI really worth of further investigations?

I hope I've made it clear that no, it isn't.