Opened 6 years ago

#8114 defect new

ThreadPool.threads leak

Reported by: Gavin Panella Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: Branch:
Author:

Description

Threads that are stopped are not removed from the TheadPool.threads list. This prevents garbage collection of the thread instances themselves and (possibly; not confirmed) thread-local storage. Programs that make frequent use of adjustPoolSize may suffer.

I've observed this in 15.4.0 (on Ubuntu 15.10 Wily) but not in 13.2.0 (on Ubuntu 14.04 Trusty).

The following can be used to reproduce the issue:

from pprint import pprint
from time import sleep

from twisted import __version__
from twisted.python import threadpool


print("Twisted " + __version__)


def count_running_threads(tp):
    return sum(1 for thread in tp.threads if thread.is_alive())


def wait_for_pool_to_adjust(tp, num_running):
    while count_running_threads(tp) != num_running:
        sleep(0.05)  # Allow threads to start or stop.
    assert len(tp.threads) == num_running, (
        "ThreadPool has %d threads, but %d were "
        "expected." % (len(tp.threads), num_running))


tp = threadpool.ThreadPool(name="foobar")
tp.start()
try:
    wait_for_pool_to_adjust(tp, 5)
    tp.adjustPoolsize(minthreads=10)
    wait_for_pool_to_adjust(tp, 10)
    tp.adjustPoolsize(minthreads=5, maxthreads=5)
    wait_for_pool_to_adjust(tp, 5)
except AssertionError:
    pprint(tp.threads)
    raise
finally:
    tp.stop()

Example from 15.4.0:

Twisted 15.4.0
[<Thread(PoolThread-foobar-0, started 140157835495168)>,
 <Thread(PoolThread-foobar-1, started 140157827102464)>,
 <Thread(PoolThread-foobar-2, stopped 140157818709760)>,
 <Thread(PoolThread-foobar-3, stopped 140157600659200)>,
 <Thread(PoolThread-foobar-4, stopped 140157592266496)>,
 <Thread(PoolThread-foobar-5, started 140157583873792)>,
 <Thread(PoolThread-foobar-6, started 140157575481088)>,
 <Thread(PoolThread-foobar-7, stopped 140157567088384)>,
 <Thread(PoolThread-foobar-8, started 140157558695680)>,
 <Thread(PoolThread-foobar-9, stopped 140157550302976)>]
Traceback (most recent call last):
  File "tpleak.py", line 29, in <module>
    wait_for_pool_to_adjust(tp, 5)
  File "tpleak.py", line 19, in wait_for_pool_to_adjust
    "expected." % (len(tp.threads), num_running))
AssertionError: ThreadPool has 10 threads, but 5 were expected.

Note that 5 threads are stopped.

Change History (0)

Note: See TracTickets for help on using tickets.