[Twisted-Python] twisted thread pool

[Glyph Lefkowitz]

> On Mar 2, 2015, at 10:36 AM, Al Niessner <al.niessner at gmail.com> wrote:
> 
> 
> Platform:
> CPU: Intel (2x with 8 core each then hyper-threading gets us to 32 virtual)
> Ubuntu 14.04
> Python 3.4 (yes, I know)
> Twisted 13.2.0
> 
> I use the vanilla twisted.internet.reactor.run() with the factory twisted.web.server.Site(root). It all runs just fine (seemingly) but,when I was snooping around on my machine, I had a ton of threads lying about. After cleaning up, I restarted my twisted application and found that I have 151 python3 threads lying about. I doubled checked and I have just twisted application running.
> 
> I browsed the documentation and the twisted thread pool claims to default between 5 and 10. However, I do abuse the deferToThread() cannot find any simple documentation on how that might feed back to the 151 threads. I thought maybe I need to exit the deferred returned from deferToThread() in some special way to clean it all up. However, I can find no documentation for it.
> 
> I have been googling about with many random search terms trying to puzzle the 151 threads from the documentation that tells me I should expect 5 to 10.
> 
> Can anyone please help understand why I have 151 threads and where they are coming from? Luckily they are all idle so the machine runs just fine.
> 
> The best story I can come up with is that there is one master thread on core-0 and then I get 10 threads in the pool for each of the other physical cores. However, the story is just numerology.

There are some bugs in the thread pool implementation (hopefully I'll get around to finishing my fix for soon, see <https://twistedmatrix.com/trac/ticket/2673 <https://twistedmatrix.com/trac/ticket/2673>> for details) which may cause this issue, but given how far out of whack those numbers are, my inclination is to think it's something else in your environment.

Can you put your full code online somewhere so that we can test it?  Ideally after fixing it to be a minimal reproducer? :)

-g

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20150302/aeae23c0/attachment-0002.html>