[Twisted-web] treq requests failing with twisted.web._newclient.ResponseNeverReceived

[Guido Winkelmann]

Hi,

Recently, I have been experiencing some problems in our codebase wth treq 
requests failing with error messages like this:

2015-04-02 19:20:22 ERROR    - pf.agent.http.client - POST 
http://127.0.0.1:5000/api/v1/agents/a113b5a8-9822-413b-b1fd-fce3014956cf has 
failed (uid: 78c7ec2a4b4d):
Traceback (most recent call last):
Failure: twisted.web._newclient.ResponseNeverReceived: 
[<twisted.python.failure.Failure <class 
'twisted.internet.error.ConnectionDone'>>]

Unfortunately, I have not been able to successfully produce a reduced test 
case that will reproduce this behaviour.  It only happens under some weird set 
of circumstances, and I am still not sure exactly which.  It appears to be 
highly sensitive to the order of requests.  Funnily enough, this error does 
not show in the codebase as it is on github right now, but if I reorder some 
of the requests, it will show.  Here is one example that has cost me some time 
and head-scratching:

https://github.com/pyfarm/pyfarm-agent/issues/249

I think that these problems have started after upgrading to Twisted 15.0, but 
I'm not completely certain about this.

Here is one of the methods that's affected by this:

https://github.com/pyfarm/pyfarm-agent/blob/master/pyfarm/agent/service.py#L231

(post_direct() is just a thin wrapper over treq.request that adds some headers 
we need in our codebase.  I have tried just using treq.request directly 
instead, the result was the same.)

Can someone tell me under which circumstances treq.request would produce an 
error like this?

Without actually knowing the source in question, I would guess that something 
in there is, for some some reason, erroneously trying to reuse a TCP 
connection that has already been closed.

Regards,

	Guido W.