[Twisted-Python] Need some enlightenment on using web client properly, or maybe nudge a bug to get fixed

[Jarosław Fedewicz]

I have written a simple service which takes data from network, massages it
until it's useful enough, and sends the results out periodically via HTTP
to an API.

It all works for a while, then I get an error like this approximately 40
minutes into the service's uptime:

ResponseNeverReceived: [<twisted.python.failure.Failure
OpenSSL.SSL.ZeroReturnError: >]


Then a couple more like this:

ResponseNeverReceived: [<twisted.python.failure.Failure
twisted.internet.error.ConnectionLost: Connection to the other side was
lost in a non-clean fashion: Connection lost.>]


Then it ends with

TimeoutError: User timeout caused connection failure.


Then every request results in the same TimeoutError. I don't know if using
HTTPS important in this case.

Restarting the whole service, of course, makes the problem go for a while.
The other side is the Slack API, so I rather assume it's not very much to
blame, it can be demonstrated to work rather reliably, all its criticisms
notwithstanding.

I cannot yet tell if this bug is a function of uptime, or the number of
requests made.

I have tried to work around the problem by discarding the agent object, and
using an HTTPConnectionPool with persistent=False, but it didn't help at
all. I think it made the problem worse because the framework seems to refer
to some objects the Agent creates, and the process becomes a CPU hogs in a
couple hours (with the TimeoutErrors still happening all the time).

The closest I've got on the internets which describes a similar problem,
apart from people complaining on StackOverflow about precisely this to
happen when they are using Scrapy, is this blog post from almost a decade
ago:
http://www.chris-wong.net/twisted-web-framework-user-timeout-caused-connection-failure/
.

There could be a small chance I'm holding it wrong(tm), but maybe there
exists a ticket, just worded differently, which could help me get to the
bottom of it.

-- 
Yaroslav Fedevych
IT Philosopher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20190702/2ffe0fe0/attachment.html>