[Twisted-Python] Twisted WebSockets Branch

[Rob Oakes]

Dear Twisted Developers,

As a quick introduction, my name is Rob Oakes. I'm the lead developer 
for a company called Guru Labs. I'm writing for two reasons:

# Reason 1: Thank You

First and foremost, I'm writing to express thanks for creating an 
excellent framework. We use Twisted extensively in our infrastructure, 
and it is typically our go-to tool for any sort of custom server.

The other two reasons have to do with some questions about the 
development of Twisted. Before I dive into those, however, let me 
provide some background.

For the better part of a year, we've been using some code out of the 
websockets (twisted.web.websockets.WebsocketsResource) branch to wrap a 
trio of custom protocols we use in one of our web based products. I 
know that the code is still pre-release, but we've generally found it 
to be stable and work very well.

# Reason Two: Websockets Development

This brings me to the second reason I'm writing. Over the past year, it 
seems as though the development on the Twisted websockets branch has 
stalled. We would like to unstall it. For this reason, Guru Labs would 
be interested in:

1.) Contracting with one of the core Twisted devs of Twisted who might 
be interested in finishing the implementation of the websockets 
wrappers (resolving the issues described in ticket 4173) or another 
party. We are happy to pay hourly rates, set a bounty, make a donation 
to the Twisted project ... whatever.

A stable implementation of websockets available in Twisted Web is a 
priority to us, and if it's a matter of money, we are happy to throw 
money where it might be needed. (If this of interest, please contact me 
off-list at roakes at gurulabs.com.)

or

2.) Assigning a Guru Labs developer (probably me) who might complete 
the work.

I've been studying the issues which are still open (specifically 4173), 
and the various branches associated with websocket development 
(websocket-4173-3 and websocket-4173-4) and it seems that most of the 
major concerns are related to the manner in which websocket connections 
are closed.

Assuming that someone else doesn't step forward ... I've already merged 
the most recent version of trunk with these branches, but I've found 
myself with several questions on how to best continue with the work.

a. Which of the two websocket branches should be used as the basis for 
new work?

On GitHub, websocket-4173-4 is marked as closed with a note telling the 
contributor to see the contribution guidelines. There are also several 
additions to the code which do not follow Twisted conventions (as I 
understand them). The last set of commits also seem to come from 
approximately the same time.

websocket-4173-4 includes code which attempts to resolve issues noted 
in 4173 that is not present in 4173-3, but there aren't really any 
comments to determine if this should be incorporated or not. At this 
point, I'm really not familiar enough with the code to draw my own 
conclusion.

b. In general, the protocol wrapper seems to work quite well. However 
when merged with the most recent version of trunk (post twisted 14), 
I've been seeing frequent  unhandled deferred errors. The most common 
is:

"twisted.internet.error.ConnectionLost: Connection to the other side 
was lost in a non-clean fashion: Connection lost"

It happens when:

* Connections are closed from the server, using transport.loseConnection
* Connections do not transmit a "close" frame prior to disconnecting
* To reiterate, this issue only started appearing after merging the 
websocket code with Twisted 14. The previous version of Twisted we were 
using (Twisted 12.3 worked flawlessly.)

No error is raised if the client correctly closes the connection or 
when using non-browser based clients (like the Python ws4py websocket 
client).

We mostly see the exception when the objects are garbage collected 
(based upon the deferred documentation at 
http://twistedmatrix.com/documents/14.0.0/core/howto/defer.html). Put 
another way, we see a whole string of errors upon stopping the reactor. 
Despite the exception, we don't see any errors in the browser client.

Also somewhat frustratingly, the traceback isn't terribly helpful. This 
is a fairly routine example:

```error
Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionLost: Connection to the other 
side was lost in a non-clean fashion: Connection lost.
```

Technically, the error is probably appropriate, as they appear when the 
connection is lost in a non-clean fashion. I am a little concerned, 
though, in that I haven't found a good way to catch or suppress the 
error. Moreover, it doesn't seem like the defferreds are getting 
garbage collected which seems like a memory leak waiting to happen.

Can someone clarify if this is:

* intended behavior, and if so, what might be a strategy I can use for 
managing the error in my wrapped protocols
* which part of the websocket code I should be looking at in order to 
try and fix the issue

Thoughts would be greatly appreciated.

Cheers,

Rob Oakes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140822/b4dede97/attachment.html>