[Twisted-web] treq requests failing with twisted.web._newclient.ResponseNeverReceived

[Guido Winkelmann]

An update on the situation:

I tried downgrading both Twisted and treq to <15.0. The problem remained, so 
this issue is /not/ new in 15.0.

When I remove the "pool" parameter in this line in treq/api.py:113:

agent = Agent(reactor, pool=pool)

the problem completely disappears, so I'm assuming the problems lies somewhere 
in the connection pooling logic.

The web server I am using in these tests is flask's built-in standalone 
webserver (http://flask.pocoo.org/). This webserver speaks HTTP 1.1, but does 
not support keep-alive (at least not with non-static content), which is 
technically allowed but may be a slightly unusual combination these days.

Regards,
	Guido W.

On Thursday 02 April 2015 19:49:59 Guido Winkelmann wrote:
>Hi,
>
>Recently, I have been experiencing some problems in our codebase wth treq
>requests failing with error messages like this:
>
>2015-04-02 19:20:22 ERROR    - pf.agent.http.client - POST
>http://127.0.0.1:5000/api/v1/agents/a113b5a8-9822-413b-b1fd-fce3014956cf has
>failed (uid: 78c7ec2a4b4d):
>Traceback (most recent call last):
>Failure: twisted.web._newclient.ResponseNeverReceived:
>[<twisted.python.failure.Failure <class
>'twisted.internet.error.ConnectionDone'>>]
>
>Unfortunately, I have not been able to successfully produce a reduced test
>case that will reproduce this behaviour.  It only happens under some weird
>set of circumstances, and I am still not sure exactly which.  It appears to
>be highly sensitive to the order of requests.  Funnily enough, this error
>does not show in the codebase as it is on github right now, but if I reorder
>some of the requests, it will show.  Here is one example that has cost me
>some time and head-scratching:
>
>https://github.com/pyfarm/pyfarm-agent/issues/249
>
>I think that these problems have started after upgrading to Twisted 15.0, but
>I'm not completely certain about this.
>
>Here is one of the methods that's affected by this:
>
>https://github.com/pyfarm/pyfarm-agent/blob/master/pyfarm/agent/service.py#L2
>31
>
>(post_direct() is just a thin wrapper over treq.request that adds some
>headers we need in our codebase.  I have tried just using treq.request
>directly instead, the result was the same.)
>
>Can someone tell me under which circumstances treq.request would produce an
>error like this?
>
>Without actually knowing the source in question, I would guess that something
>in there is, for some some reason, erroneously trying to reuse a TCP
>connection that has already been closed.
>
>Regards,
>
>	Guido W.
>
>_______________________________________________
>Twisted-web mailing list
>Twisted-web at twistedmatrix.com
>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web