[Twisted-web] Stuck twisted-web2 worker threads after client closes socket after sending data < content length of HTTP Post

[Kamal Choudhary]

On my app server which is built upon twisted+web2 I have seen that at times
some of worker threads will get blocked and won't get released ever.
Eventually my app server runs out of worker threads and starts throwing
message "503 Service not available"

On more debugging I have found that I can easily recreate this situation by
doing a incomplete HTTP Post with data less then content length specified in
POST header. After this incomplete HTTP Post with lesser bytes of data if I
close the client socket or kill client process or reboot client machine, in
all these three cases it results in a hanged thread on my app server. This
hanged threads remains on app server for ever till kill app server process,
no time out of any kind.

Further debugging into web2 code I found that
stream.BufferedStream.readExactly doesn't come out of
stream.BufferedStrem._readUntil even if I close client side socket. After
putting some debug statements I found that self.stream.read() in
stream.BufferedStream._readUntil doesn't return None as a signal for End Of
File even after client has closed the socket connection. Since it doesn't
get None it will never return with whatever data it has received so far from
client.

To reproduce this issue, I have written a minimal twisted-web2 server and a
client that does a HTTP Post to it with incomplete data. After Post it
closes the client socket and sleeps for sometime. Finally it prints the
stack of all process threads, which clearly shows one blocked twisted worker
thread. In the same program if you change it to do complete HTTP Post, in
the thread dump there is no blocked twisted worker thread.

Here is the stack traceback of blocked twisted worker thread.

  File "C:\Python24\lib\threading.py", line 442, in __bootstrap
    self.run()

  File "C:\Python24\lib\threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)

  File
"/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/threadp=
ool.py",
line 161, in _worker

  File
"/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/context=
.py",
line 59, in callWithContext

  File
"/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/context=
.py",
line 37, in callWithContext

  File
"/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/web2/wsgi.py",
line 190, in run

  File "C:\work\twisted-head\twisted_server3.py", line 18, in dummy_app
    out_content =3D environ['wsgi.input'].read(int(environ['CONTENT_LENGTH'=
]))

  File
"/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/web2/wsgi.py",
line 74, in read

  File
"/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/internet/threa=
ds.py",
line 81, in blockingCallFromThread

  File "C:\Python24\lib\Queue.py", line 119, in get
    self.not_empty.wait()

  File "C:\Python24\lib\threading.py", line 203, in wait
    waiter.acquire()


To run this program twisted, twisted.web2, zope and threadframe modules are
required.



import sys, time, socket
from threading import Thread
import threadframe, traceback
from twisted.web2 import server, channel, static, wsgi, resource
from twisted.internet import reactor

class TwistedWebServerThread(Thread):

    def __init__(self, app):
        Thread.__init__(self, name=3D"Twisted")
        factory =3D channel.HTTPFactory(server.Site(wsgi.WSGIResource(app)))
        reactor.listenTCP(8080, factory, interface=3D"0.0.0.0")

    def run(self):
        reactor.run(installSignalHandlers=3DFalse)

def dummy_app(environ, start_response):
    out_content =3D environ['wsgi.input'].read(int(environ['CONTENT_LENGTH'=
]))
    start_response('200 OK', [('Content-type', 'text/plain')])
    return [out_content]

#data intentionally less then content length
D =3D "POST /cgi-bin/minapi.py HTTP/1.0\r\n"+ \
 "Host: 127.0.0.1:8080\r\n"+\
 "Content-Type: text/xml\r\n"+\
 "Content-Length: 152\r\n"+\
 "\r\n"+\
 "<?xml version=3D'1.0'?>\n"+\
 "<methodCall>\n"+\
 "<methodName>getStateName</methodName>\n" +\
 "<params>\n" +\
 "<param>\n" +\
 "<value><int>41</int></value>\n" #+\
# "</param>\n" +\
# "</params>\n" +\
# "</methodCall>\n"

#D =3D open("data.txt3", "rb").read()

def half_post():
    s =3D socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect(("127.0.0.1", 8080))
    s.send(D)
    s.close()

def write_server_threads():
    frames =3D threadframe.threadframe()
    for frame in frames:
        print '-' * 72
        for linestr in traceback.format_stack(frame):
            print linestr

if __name__ =3D=3D "__main__":
    s =3D TwistedWebServerThread(dummy_app)
    s.start()
    time.sleep(0.2)
    half_post()
    time.sleep(5)
    write_server_threads()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-web/attachments/20080313/52=
79c354/attachment.htm