Twisted sometimes can't create the waker on Windows when AVG is installed and Active Surf Shield is enabled

[glyph]

Sometimes, rarely, on certain Windows installations, users see this error:

Traceback (most recent call last):

  File "whatever.py", line 1, in <module>

    from twisted.internet import reactor

  File "twisted\internet\reactor.py", line 38, in <module>

    selectreactor.install()

  File "twisted\internet\selectreactor.py", line 200, in install

    reactor = SelectReactor()

  File "twisted\internet\selectreactor.py", line 72, in __init__

    posixbase.PosixReactorBase.__init__(self)

  File "twisted\internet\posixbase.py", line 170, in __init__

    ReactorBase.__init__(self)

  File "twisted\internet\base.py", line 446, in __init__

    self._initThreads()

  File "twisted\internet\base.py", line 869, in _initThreads

    self.installWaker()

  File "twisted\internet\posixbase.py", line 206, in installWaker

    self.waker = _Waker(self)

  File "twisted\internet\posixbase.py", line 77, in __init__

    client.connect(server.getsockname())

  File "<string>", line 1, in connect

socket.error: [Errno 10049] The requested address is not valid in its context

This occurs because socket.getsockname() spuriously returns "0.0.0.0" for the host part of the address, despite the fact that it was explicitly bound to "127.0.0.1".

Possibly related tickets:

• #3748
• #1369

[glyph]

On IRC, some users are reporting that this may have something to do with the AVG virus scanner. One user who had it installed ("Ballig"), but had the firewall disabled, was experiencing the problem. Hopefully they can fill in some more details here.

No amount of stress-testing socket.getsockname() causes the problem to happen for me on a vanilla Windows XP installation.

[glyph]

I installed  AVG Free and the problem immediately manifested.

I uninstalled and it went away.

Specifically, I disabled "Active Surf Shield" and I could make the problem start and stop on demand in a running Python process. (Open the "advanced settings" UI, click on "LinkScanner", check/uncheck "Active Surf Shield" and hit "Apply".)

Some data points:

• the host is occasionally wrong (it's "0.0.0.0" approximately 1 out of every 3000 sockets, but the number is definitely non-deterministic) but the port number seems to remain correct.
• the behavior is an intermittent failure of bind(), not of getsockname(); if a socket erroneously reports its address as "0.0.0.0", it will continue to do so forever.

Someone should report this problem to AVG, but given that probably a zillion people have this installed on their computers and inevitably don't know how to upgrade, I think we should work around the buggy behavior by always passing "127.0.0.1" for the host.

[pahan]

(In [26924]) Branching to 'windows-getsockname-3831'

[PenguinOfDoom]

I attached a minimal example. If you run it, you might observe that the failing port is always 1028.

Port 1028 is part of some heinous undocumented ninja DCOM shit which might or might have disappeared before Windows XP. Naturally, there are exploits that use this port. AVG, in its infinite kindness, protects port 1028.

[PenguinOfDoom]

minimal example

[itamar]

If we are certain 0.0.0.0 means 127.0.0.1, we should also fix this for transport.getHost().

[exarkun]

Replying to glyph:

I installed  AVG Free and the problem immediately manifested.

What test were you performing?

[exarkun]

Replying to PenguinOfDoom:

I attached a minimal example. If you run it, you might observe that the failing port is always 1028.

Hmm. This never fails on my machine, either with AVG or without it, either with LinkScanner enabled or disabled.

[exarkun]

After jiggling some things around, the minimal example did start to fail, but the failing port jumps around a lot; it is not limited to 1028. Still, this seems to demonstrate the misbehavior pretty well. After reverting to a system snapshot taken prior to AVG installation, the example stopped failing.

If we are certain 0.0.0.0 means 127.0.0.1, we should also fix this for transport.getHost().

I'm not sure we're certain of that. I don't think anyone has tested other addresses.

[PenguinOfDoom]

Replying to exarkun:

If we are certain 0.0.0.0 means 127.0.0.1, we should also fix this for transport.getHost().

I'm not sure we're certain of that. I don't think anyone has tested other addresses.

I did! No matter what address you pick, getsockname() returns 0.0.0.0 (as long as you choose or get assigned a broken port). It also returns 0.0.0.0 if you bound to 0.0.0.0 to begin with :)

[glyph]

1. The test docstring really needs an explanation of why we would want to ignore the getsockname result. If I didn't know about this ticket and I saw that test, I would probably think it was added mistakenly, and "fix" it to pay attention to getsockname (unless I noticed the comment in the implementation, of course).
2. The patching that the test does is overbroad and might have some bad consequences if any other Python program attempts to do something with sockets while this test is running. At the very least I would recommend patching posixbase rather than socket; better would be be to move the waker code to its own module and patch that; even better would be to patch only the _Win32Waker class.

[exarkun]

Don't forget that functions can take parameters too. We don't always have to use patch.

[thijs]

Replying to glyph:

I installed  AVG Free and the problem immediately manifested. I uninstalled and it went away.

Which version of AVG Free was that? Maybe it's fixed in more recent versions?

[PenguinOfDoom]

Seems to work fine with AVG 9.0. Hooray.

[Szumo]

Definitely appears with AVG 8.5 though.