[Twisted-Python] twisted thread pool

Al Niessner al.niessner at gmail.com
Wed Mar 4 09:59:00 MST 2015


Nevermind. While building the test case I figured out where I went wrong. I
had a subprocess pool that I forgot to close and it just grew out of hand.

I checked and twisted is running at 4 processes. Sounds better.

Thanks for all of the help.

On Mon, Mar 2, 2015 at 1:50 PM, Glyph Lefkowitz <glyph at twistedmatrix.com>
wrote:

> On Mar 2, 2015, at 10:36 AM, Al Niessner <al.niessner at gmail.com> wrote:
>
>
> Platform:
> CPU: Intel (2x with 8 core each then hyper-threading gets us to 32 virtual)
> Ubuntu 14.04
> Python 3.4 (yes, I know)
> Twisted 13.2.0
>
>
> I use the vanilla twisted.internet.reactor.run() with the
> factory twisted.web.server.Site(root). It all runs just fine (seemingly)
> but,when I was snooping around on my machine, I had a ton of threads lying
> about. After cleaning up, I restarted my twisted application and found that
> I have 151 python3 threads lying about. I doubled checked and I have just
> twisted application running.
>
> I browsed the documentation and the twisted thread pool claims to default
> between 5 and 10. However, I do abuse the deferToThread() cannot find any
> simple documentation on how that might feed back to the 151 threads. I
> thought maybe I need to exit the deferred returned from deferToThread() in
> some special way to clean it all up. However, I can find no documentation
> for it.
>
> I have been googling about with many random search terms trying to puzzle
> the 151 threads from the documentation that tells me I should expect 5 to
> 10.
>
> Can anyone please help understand why I have 151 threads and where they
> are coming from? Luckily they are all idle so the machine runs just fine.
>
> The best story I can come up with is that there is one master thread on
> core-0 and then I get 10 threads in the pool for each of the other physical
> cores. However, the story is just numerology.
>
>
> There are some bugs in the thread pool implementation (hopefully I'll get
> around to finishing my fix for soon, see <
> https://twistedmatrix.com/trac/ticket/2673> for details) which may cause
> this issue, but given how far out of whack those numbers are, my
> inclination is to think it's something else in your environment.
>
> Can you put your full code online somewhere so that we can test it?
> Ideally after fixing it to be a minimal reproducer? :)
>
> -g
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20150304/4acf792f/attachment-0002.html>


More information about the Twisted-Python mailing list