[Twisted-Python] data dispatch on massive connection counts

Tobias Oberstein tobias.oberstein at tavendo.de
Mon Nov 14 12:57:46 EST 2011


> That's definitely not a limit on Python or Twisted. That's an OS limit.
> Are you opening lots of files in addition to your sockets?

As it turns out, you are right, and the full answer is most unpleasant:

This

http://www.freebsd.org/cgi/query-pr.cgi?pr=148581&cat=

bug in FreeBSD libc, which is there in i386/amd64 up to and including FreeBSD 9 RC1

and the fact that Python uses fopen() from libc, not open() from Posix

results in the situation that you can't have more than 32k FDs done.

In my situation, it's like this: the new kqueue reactor will happily accept
50k TCPs .. no problem, since Python isn't involved with fopen() here.

But as soon as a Python open() and thus fopen() happens, the new FD
would need to be >32k and that does not work, since the braindead libc
on FreeBSD defines fileno to be SHORT_MAX.

Now I'm running out of options.

I was told that the new Python 3 IO system does not use fopen(),
however Twisted is not yet there on Python 3, right?

There is a backport of that new IO to Python 2.7, but I'm not sure
if thats transparent for calls like Python open().

I can't open first i.e. 100 files, since the set of files needed is not
fixed in the beginning ..i.e when Twisted does a log file switch.

Well, this is all absolutely sad. Now we (nearly) have new kqueue,
it does fly, but I can't break above 32k anyway ..



More information about the Twisted-Python mailing list