[Twisted-Python] No connectionLost

Brian Warner warner at lothar.com
Fri Dec 5 19:38:06 EST 2003


> Is this expected behavior?  What can I do to avoid this problem?

Yes. Twisted applications (indeed all endpoints communicating over
packet-switched networks) have no way to know about connections that are
broken somewhere in the middle. If the remote program is terminated, their
kernel will probably send you a TCP FIN to shutdown the connection. If your
end terminates then you know your connection is lost. But the only way to
detect that something in the middle has gone away is to notice that the
remote system is not getting your packets.

The key thing to pay attention to is that there is no "connection" as such.
There are two endpoints which may or may not hold the same beliefs about
their ability to communicate, and any individual packet may or may not arrive
at its destination.

The standard technique to deal with this (and try to maintain the illusion of
"connection") this is with pings that must be acknowleged. The default TCP
socket usually shuts down after transmitted data has been outstanding
(unacknowleged) for a while (10s of minutes? maybe an hour?), but if there is
no data being transmitted then you may never discover that the connection is
no longer useable.

You can turn on TCP Keepalives with a socket option, which causes a probe to
be sent after some period of inactivity (two hours on my system). When this
probe goes unacknowleged for a while (same as regular data), the connection
is declared dead. This will cause the same kind of connectionLost
notification as happens when the remote program terminates and their kernel
tells you about the socket being closed.

You can add code to your protocol to make additional pings on a regular
basis, and then establish timeouts which manually drop the connection if they
are not acknowleged in time. You will need to make a tradeoff between extra
traffic and the latency that will go by before you detect the connection has
gone away (to detect a lost connection within 60 seconds, you must send some
data every 60 seconds). With shorter timeouts you also run the risk of
false-shutdowns, where the app overreacts to a temporary network outage.

Practically speaking, most connections run over fairly reliable networks, and
programs crash/quit much more frequently than intermediate routers. So the
"connection lost" FIN packet is usually good enough. With a VPN providing
additional routing complexity in the middle, that assumption doesn't hold up
so well.

hope that helps,
 -Brian




More information about the Twisted-Python mailing list