[Twisted-web] web2.client pipelining (was: Web2: What is to be Done)

[Scott Lamb]

I hadn't noticed until now that y'all have added a  
twisted.web2.client to trunk. (Copied from a web2-client4 branch; I  
guess the previous castle burned down, fell over, and then sank into  
the swamp. ;)

How does the pipelining work? I ask because I've grown quite picky  
about the closing states of HTTP. I'm almost certainly preaching to  
the choir, but I'll quote RFC 2616 section 8.1.2.2:

     Clients which assume persistent connections and pipeline  
immediately
     after connection establishment SHOULD be prepared to retry their
     connection if the first pipelined attempt fails. If a client does
     such a retry, it MUST NOT pipeline before it knows the  
connection is
     persistent. Clients MUST also be prepared to resend their  
requests if
     the server closes the connection before sending all of the
     corresponding responses.

and...umm, I can't find the RFCs, but some combination says that:

(A) the kernel's TCP stack should immediately dump all its buffers  
and abort on a RST, so effectively the application gets a ECONNRESET  
sent back from the future
(B) the sequence the HTTP server takes when receiving a request after  
a FIN causes a RST to be sent

So when using pipelining, the client must be prepared for this case:

    1. send request A
    2. send request B
    3. receive an ECONNRESET...and not even know if the webserver  
received or handled request A, much less what the response might have  
been

For non-idempotent HTTP requests (a lot of SOAP calls), that  
ambiguity is not acceptable, so I need a way to toggle pipelining.  
Looks like that's this?

     p = HTTPClientProtocol()
     p.readPersistent = PERSIST_(NO_)?PIPELINE

Second, if it is using pipelining, it has to do the retry. That part  
I don't see. Is the caller expected to do it? twisted/web2/client/ 
http.py has this:

     def connectionLost(self, reason):
         self.readPersistent = False
         self.setTimeout(None)
         self.manager.clientGone(self)
         # Tell all requests to abort.
         for request in self.inRequests:
             if request is not None:
                 request.connectionLost(reason)

Warm regards,
Scott

-- 
Scott Lamb <http://www.slamb.org/>