[Twisted-web] web2.client keep-alive problem
Scott Lamb
slamb at slamb.org
Fri Oct 21 22:51:53 MDT 2005
I've run into another problem in my application that uses the dreid/
web2-client branch. This one I'm not sure how to solve, or even
kludge around for the moment.
My application makes a bunch of HTTP requests in serial. It needs to
see the previous response to make the next request.
When I handle a HTTP response, I originally wrote it like this
(simplified):
def _makeRequest(self, ...):
req = client.Request(...)
req.addCallback(self._handleResponse)
def _handleResponse(self, response):
stream.readStream(response.stream, self._handleResponseBody)
def _handleResponseBody(self, body):
self._makeRequest(...)
This worked fine - proper HTTP Keep-Alives and whatnot - until I got
a large response back. Then I realized that stream.readStream gives
me chunks and I should instead be using this (if I want to handle the
whole response at once):
def _handleResponse(self, response):
self.body = ''
stream.readStream(response.stream, self._handleResponseChunk
).addCallback(self._handleResponseBody)
def _handleResponseChunk(self, chunk):
self.body += chunk
def _handleResponseBody(self):
self._makeRequest(...)
self.body = None
This works with the chunking, but I am now making an HTTP connection
for each request. HTTPClient._endResponse() calls
self.response.stream.finish() and then tears down the connection if
no request is pending. But stream.finish() doesn't call my deferred
right away; it happens later from the reactor. So though I want to
make another request, I haven't had my chance yet. My connection is
torn down, then I make a new one and do an expensive SSL handshake,
which is all wasted round trips, bandwidth, and CPU time.
I can think of two ways to solve this:
(1) make stream.ProducerStream.finish() call _handleResponseBody
right away. But there's a "reactor.callLater(0, self._read)" in
stream._StreamReader that looks to be there for a reason. I haven't
thought it through, but I suspect there are unsolvable recursion
problems with this approach.
(2) make HTTPClient not check if the connection should be torn down
until after my deferred has had a chance to run. Would just a
"reactor.callLater(0, self._endResponse2)" be enough? Are callLaters
guaranteed to be run in order?
--
Scott Lamb <http://www.slamb.org/>
More information about the Twisted-web
mailing list