[Twisted-web] Response hanging after headers sent

[exarkun at twistedmatrix.com]

On 02:27 pm, bruno at distributedmatter.net wrote:
>Hello,
>
>I've been trying to implement a simple notification system via long 
>polling
>using Twisted 14.0.0. On the client side, a web page makes Ajax calls 
>to an
>"update" resource, which will deliver the latest status message known 
>to
>the application. On the server side, a thread feeds a queue with status
>events and another one polls this queue and sends the latest value to 
>any
>pending request, trying to keep the request open for 30 seconds, or 
>until a
>new status event arrives, whichever happens sooner (this is the long
>polling).
>
>[snip]
>
>class CounterUpdateResource(Resource):
>    def __init__(self, msg_queue):
>        Resource.__init__(self)
>        self.msg_queue = msg_queue
>        self.msg = {}
>        self.etag = None
>        self.pending_requests = []
>
>        t = threading.Thread(target=self._poll_messages)
>        t.daemon = True
>        t.start()
>
>    def _poll_messages(self):
>        while True:
>            try:
>                self.msg = self.msg_queue.get(True, 30)
>                self.etag = '"%s"' % (uuid.uuid4(),)
>                print "-> Got new message: %s" % (self.msg,)
>            except Queue.Empty:
>                pass
>            json_str = json.dumps(self.msg)
>            json_len = len(json_str)
>            while True:
>                try:
>                    request = self.pending_requests.pop()
>                    self._actual_render(request, json_str, json_len)
>                except IndexError:
>                    break
>
>    def _actual_render(self, request, json_msg, json_len):
>        try:
>            request.setETag(self.etag)
>            request.setHeader("Content-Type", "application/json")
>            request.setHeader("Content-Length", json_len)
>            request.write(json_msg)
>            request.finish()
>        except:
>            pass

`_poll_messages` is run in a thread.  It calls `_actual_render` - so 
`_actual_render` is run in a thread.  It calls these `request` methods - 
so they are run in a thread.

Twisted APIs are not thread-safe - except for one or two that are 
explicitly marked as being thread-safe.

Since this code calls Twisted APIs (the request methods) in a thread, 
they have undefined behavior.

If you change this so that `_actual_render` (or all of the request 
methods) are called in the reactor thread then I expect it will start to 
work reliably.

Stepping back, you might consider getting rid of the multithreaded in- 
process polling and instead make your application fully event-driven. 
Instead of having one part of your program put messages into a queue and 
another part look at the queue from time to time and do work if it finds 
an item, have one part of your program call a method implemented by 
another part of your program which does the work.  No threads, no 
polling, hooray.

Stepping back even further, there are existing libraries for doing this 
kind of thing with Twisted already.  There's Divmod Nevow's Athena and 
there's Minerva.  There are also other approaches to "real time" web - 
such as websockets (see Autobahn and txwebsocket).  You may not actually 
have to implement this low-level plumbing yourself.

Jean-Paul