[Twisted-web] HTTP client TODOs
slamb at slamb.org
Fri Nov 4 10:47:14 MST 2005
On Nov 4, 2005, at 8:39 AM, dreid at dreid.org wrote:
> Ok, time to clear things up about the web2-client. Do not count on
> branches/dreid/web2-client/ It isn't going to be maintained. Why?
> it is full of kludges, doesn't even do http/1.1 properly and is
> known to
> break intermittently.
It doesn't do HTTP/1.1 properly? What's wrong?
> dreid/web2-client was created for my employer, it
> works in the confines of what we're doing, which isn't very much
> http, but
> we still needed something that worked differently than
> I feel iffy about that fix, but I've always felt iffy about the
> way it
> creates a new request object. I'll let foom comment on the changes
> to stream.
The POST stream fix? Yeah, I know that one's crap. But it works for
me. My patch includes a hyperlink to the discussion of a better fix,
but I haven't had time to actually do it.
> And the AlreadyCalled exception is a known bug, we've
> encountered it maybe twice at work, in the 3 or so months that we've
> been using the branch of the branch. I haven't been able to reproduce
> it reliably enough to fix it.
For a while, I had it happening consistently. Maybe I can get that to
> So yes, dreid/web2-client ... not my best work, it's a
> lot of kludges I'm sorry for inflicting this upon you, I'm even more
> sorry that I have to tell you that it's not going to go anywhere, and
> backwards compatibility isn't going to happen.
Dude, you can apologize all you want, but I'm calling it a tremendous
success. Even if all copies of dreid/web2-client-related code were to
be annihilated tomorrow, it totally saved my ass.
Let me tell you how I used it. I had to load test our HTTP-based
protocol. I had a Java-based load tester tool that a contractor had
spent...stories vary...one guy said four months, another said six to
nine months...writing. He would tell them things like "if I just make
this change, it will compile". Ridiculously low standards. Bizarre
verbose code with no point. I had this thing dumped on me and needed
it working in a week. It took me a couple days to figure out that
there was nothing salvagable there at all.
So I looked for something else. I like Python. I'd heard of Twisted
but never used it before. I saw that twisted.web.client didn't do
Keep-Alives, which I needed. Google showed me one of your commits on
this branch, and I started coding. In a day, I had something usable.
It wasn't great, but it was a hell of a lot better than using the
contractor's code. I developed it as I needed it. Now I've got a tool
that can generate a load that is quite configurable, is reasonably
fast, is pleasant to run (nice optparse stuff; SIGINT, SIGHUP, and
SIGTERM do a nice graceful shutdown), reports quick results during
the run, and generates full reports into a relational database which
I have a web interface for and a nice graph tool. The load tester is
700 lines of code, with verbose comments.
I have absolutely no idea how I could have done the rapid development
I needed had dreid/web2-client not existed. Even if I have to
completely rewrite my tool from scratch, I needed it _right_then_.
Even if I hadn't, it was useful as a prototype.
Anyway, I don't think the situation with this branch is nearly as bad
as you say. Yes, it's kludgey. Yes, I found like 10 bugs. But it
essentially works, and I don't think the design is fundamentally
flawed. Tests can be added, and it can be refactored. Other people
like me will come along, desperately need it, and fix a bunch of
Scott Lamb <http://www.slamb.org/>
More information about the Twisted-web