[Twisted-web] overkill context switches

[Andrew Bennetts]

On Mon, Sep 26, 2005 at 05:50:24PM +0200, Andrea Arcangeli wrote:
[...]
> This should be a python or twisted-thread-pool bug. I'm not sure about the best
> way to track it down, especially because so far I can reproduce it only on the
> server, that is a bit slower and that I cannot use for debugging.

Twisted's threadpools are pretty straightforward wrappers around Python's
threading module.  See twisted.python.threadpool.

I'd expect you could reproduce this with a simple python script that starts
two threads, with one putting items on a Queue.Queue, and the other reading
them off, as a simple approximation of the threadpool arrangement.

In fact, here's a script that does this:

----
import threading
import Queue

q = Queue.Queue()

def producer():
    # Count from 0 to 99, over and over.
    i = 0
    while True:
        q.put(i)
        i += 1
        i %= 100


def consumer():
    while True:
        i = q.get()
        i * 2

p = threading.Thread(target=producer)
c = threading.Thread(target=consumer)
p.setDaemon(True)
c.setDaemon(True)
p.start()
c.start()

import time; time.sleep(10)
----

vmstat reports lines like:

 1  1 350740  18828  38260 204880    0    0  2696     0 1099 100315 83 17  0  0
 3  2 350740  16852  38336 206760    0    0  1956    20 1133 100174 86 14  0  0
 1  1 350740  15116  38444 208352    0    0  1684   592 1106 99901 85 15  0  0
 1  1 350740  12760  38520 210640    0    0  2364     0 1119 100183 87 13  0  0
 1  1 350740  11024  38612 212336    0    0  1792     0 1123 100255 85 15  0  0
 1  1 350740   9048  38696 214216    0    0  1964     0 1129 100057 85 15  0  0

(it also appears that the producer adds to the queue faster than the consumer
gets from it.  Doesn't really matter, aside from the memory cost.)

I don't see it with a simple script that has two threads running in parallel
with no inter-thread communication at all, however.  I'd say the place to
look is Python's implementation of thread synchronisation primitives.

-Andrew.