[Twisted-web] web2.client pipelining (was: Web2: What is to be Done)

[James Y Knight]

On Sep 13, 2006, at 2:00 PM, Scott Lamb wrote:
> I hadn't noticed until now that y'all have added a  
> twisted.web2.client to trunk. (Copied from a web2-client4 branch; I  
> guess the previous castle burned down, fell over, and then sank  
> into the swamp. ;)
>
> <pipelining/retry/etc>

The client that's in there right now is a low-level http client. The  
higher level web client (the connection manager, queuing, pooling,  
etc.) is not completed. That is where the retry logic and the  
determination of which requests can be pipelined and which cannot  
would go. Please note that even *keepalive* can only be (reliably)  
used for idempotent actions. For example, client sends request A,  
gets response A, sends request B, but at the same time, server closes  
the connection from a timeout condition.

The twisted.web2.client.HTTPClientProtocol.submitRequest has a  
keyword arg "closeAfter", which defaults to True. Thus, without you  
doing anything to change it, there is no keepalive and no pipelining.  
Each connection will only be usable for one request. Pipelining and  
keepalive features should really only be enabled when a request is  
submitted by a client manager which is able to handle the appropriate  
failure retry conditions.

James