[Twisted-web] HTTP client TODOs

[Scott Lamb]

Jan Alonzo wrote:
> I am new to twisted and one of my priority right now is to help out with the
> HTTP client code in the web2 module. Is there a TODO specific to the client
> library somewhere that I can look at? 

I saw some '@@@' markers in comments indicating things that need to be 
fixed or tested.

I have some patches which aren't quite ready for inclusion. No unit 
tests, plus the stream-kludgey-reset.patch is true to its name; it needs 
to be reworked in the manner described in the list discussion.

Actually, the auth-python23.patch is ready, if a committer can merge it. 
I added a unit test in the last patch, which Jp broke on Python 2.3 when 
he applied it. (Don't mess with perfection. ;)

Regards,
Scott
-------------- next part --------------
Fixes bug in which authorization retries of POST actions would not send the
actual data.

http://twistedmatrix.com/pipermail/twisted-web/2005-September/001924.html

Index: twisted/web2/stream.py
===================================================================
--- twisted/web2/stream.py	(revision 14571)
+++ twisted/web2/stream.py	(working copy)
@@ -9,7 +9,8 @@
 
 The IStream interface is very simple. It consists of two methods:
 read, and close. The read method should either return some data, None
-if there is no data left to read, or a Deferred. Close frees up any
+if there is no data left to read, or a Deferred. The reset method will
+make subsequent reads start over from the beginning. Close frees up any
 underlying resources and causes read to return None forevermore.
 
 IByteStream adds a bit more to the API:
@@ -61,6 +62,10 @@
         Errors may be indicated by exception or by a Deferred of a Failure.
         """
         
+
+    def reset():
+        """Resets the stream."""
+
     def close():
         """Prematurely close. Should also cause further reads to
         return None."""
@@ -249,16 +254,19 @@
             if len(mem) < length:
                 raise ValueError("len(mem) < start + length")
             self.length = length
+        self.written = False
 
+    def reset(self):
+        self.written = False
+
     def read(self):
-        if self.mem is None:
+        if self.mem is None or self.written:
             return None
+        self.written = True
         if self.length == 0:
             result = None
         else:
             result = buffer(self.mem, self.start, self.length)
-        self.mem = None
-        self.length = 0
         return result
 
     def close(self):
Index: twisted/web2/client/http.py
===================================================================
--- twisted/web2/client/http.py	(revision 14571)
+++ twisted/web2/client/http.py	(working copy)
@@ -52,8 +52,8 @@
     @ivar extraHeaders: Additional headers you want sent as part of the
         http request
 
-    @type body: str
-    @ivar body: The data associated with this request. This value can be None.
+    @type stream: IByteStream
+    @ivar stream: The data associated with this request. This value can be None.
 
     @type deferred: Deferred
     @ivar deferred: The C{Deferred} that will fire when this
@@ -286,7 +286,7 @@
         # Lastly, if there's a body, send that.
         if self.request.stream:
             if self.factory.logging:
-                _doLog("[Sent] body (%d byte(s))" % len(self.request.body))
+                _doLog("[Started Sending Body]")
 
             d = stream.StreamProducer(self.request.stream).beginProducing(self.transport)
             d.addCallback(self._endSendRequest)
@@ -709,6 +709,8 @@
         if authHeader and self.authHandlers:
             creds = self.authHandlers[authHeader[0].lower()].getCredentials(authHeader[1], request)
             if creds:
+                if request.stream is not None:
+                    request.stream.reset()
                 req = Request(request.method, request.uri, request.args,
                               request.headers, request.stream)
 
-------------- next part --------------
Fixes direct SSL connections. (Though you should use stunnel as a proxy,
anyway - it's much faster.)

Index: twisted/web2/client/http.py
===================================================================
--- twisted/web2/client/http.py	(revision 14553)
+++ twisted/web2/client/http.py	(working copy)
@@ -659,7 +659,7 @@
         if hasattr(self, "wrappingFactory"):
             result = reactor.connectTCP(self.host, self.port, self.wrappingFactory)
         elif self.sslContextFactory is not None:
-            result = reactor.connectSSL(self.host, self.port, self.sslContextFactory, self)
+            result = reactor.connectSSL(self.host, self.port, self, self.sslContextFactory)
         else:
             result = reactor.connectTCP(self.host, self.port, self)
 
-------------- next part --------------
Fixes a bug which caused a new HTTP connection for every request.

HTTPClient._endResponse() was calling self.response.stream.finish(), then
immediately checking if there are any more requests and, since there weren't
any, shutting down the connection. stream._StreamReader doesn't call the
completion deferred until later (with a callLater(0).) Thus, my load tester's
_handleFullResponse was never getting a chance to say "hey, I do have another
request". Give it that chance by using the same CallLater(0) trick before
deciding whether to tear down the connection.

Index: twisted/web2/client/http.py
===================================================================
--- twisted/web2/client/http.py	(revision 14553)
+++ twisted/web2/client/http.py	(working copy)
@@ -482,12 +482,18 @@
             self.state = CONNECTED
         self._decoder = None
 
-        connection = None
         if self.response is not None:
             if self.response.stream is not None:
                 self.response.stream.unregisterProducer()
                 self.response.stream.finish()
 
+        reactor.callLater(0, self._endResponse2)
+
+    def _endResponse2(self):
+        connection = None
+        if self.factory.logging:
+            _doLog("[End Response 2]")
+
         # assert(self.connection is not None)
 #             if self.request is not None and self.request.deferred is not None:
 #                 self.request.deferred.callback(self.response)
-------------- next part --------------
Makes basic authorization work on Python 2.3, which doesn't have b64encode.

Index: twisted/web2/client/auth.py
===================================================================
--- twisted/web2/client/auth.py	(revision 14553)
+++ twisted/web2/client/auth.py	(working copy)
@@ -54,7 +54,7 @@
     scheme = 'Basic'
 
     def encodeCredentials(self, creds, challenge, request=None):
-        return base64.b64encode(':'.join(creds))
+        return base64.encodestring(':'.join(creds)).strip()
 
 
 class HTTPDigestAuthHandler(BaseHTTPAuthHandler):