[Twisted-web] web2.client keep-alive problem

Scott Lamb slamb at slamb.org
Fri Oct 21 22:51:53 MDT 2005

I've run into another problem in my application that uses the dreid/ 
web2-client branch. This one I'm not sure how to solve, or even  
kludge around for the moment.

My application makes a bunch of HTTP requests in serial. It needs to  
see the previous response to make the next request.

When I handle a HTTP response, I originally wrote it like this  

     def _makeRequest(self, ...):
         req = client.Request(...)

     def _handleResponse(self, response):
         stream.readStream(response.stream, self._handleResponseBody)

     def _handleResponseBody(self, body):

This worked fine - proper HTTP Keep-Alives and whatnot - until I got  
a large response back. Then I realized that stream.readStream gives  
me chunks and I should instead be using this (if I want to handle the  
whole response at once):

     def _handleResponse(self, response):
         self.body = ''
         stream.readStream(response.stream, self._handleResponseChunk

     def _handleResponseChunk(self, chunk):
         self.body += chunk

     def _handleResponseBody(self):
         self.body = None

This works with the chunking, but I am now making an HTTP connection  
for each request. HTTPClient._endResponse() calls  
self.response.stream.finish() and then tears down the connection if  
no request is pending. But stream.finish() doesn't call my deferred  
right away; it happens later from the reactor. So though I want to  
make another request, I haven't had my chance yet. My connection is  
torn down, then I make a new one and do an expensive SSL handshake,  
which is all wasted round trips, bandwidth, and CPU time.

I can think of two ways to solve this:

(1) make stream.ProducerStream.finish() call _handleResponseBody  
right away. But there's a "reactor.callLater(0, self._read)" in  
stream._StreamReader that looks to be there for a reason. I haven't  
thought it through, but I suspect there are unsolvable recursion  
problems with this approach.

(2) make HTTPClient not check if the connection should be torn down  
until after my deferred has had a chance to run. Would just a  
"reactor.callLater(0, self._endResponse2)" be enough? Are callLaters  
guaranteed to be run in order?

Scott Lamb <http://www.slamb.org/>

More information about the Twisted-web mailing list