[Twisted-Python] data dispatch on massive connection counts

Tobias Oberstein tobias.oberstein at tavendo.de
Mon Nov 14 07:57:21 MST 2011


> > 2) Too many files.
> > Is there another limit specifically for files, and/or something tunable in
> Python/Twisted?
> 
> That's definitely not a limit on Python or Twisted. That's an OS limit.
> Are you opening lots of files in addition to your sockets?

Python open() will bail out at 32768 open files.

This despite the fact that 

a) resource.getrlimit(resource.RLIMIT_NOFILE) = 200k
b) I can accept 50k sockets

Obviously, it's at least not Twisted related ..

=========

import resource

max = resource.getrlimit(resource.RLIMIT_NOFILE)
cnt = 0
print "maximum FDs", max

fds = []

while cnt < max:
   f = open("/tmp/test_%d" % cnt, "w")
   f.write("test")
   fds.append(f)
   cnt += 1
   if cnt % 1000 == 0:
      print "opened %d files" % cnt

print "ok, created %d files" % cnt

=========

maximum FDs (200000L, 200000L)
opened 1000 files
opened 2000 files
opened 3000 files
opened 4000 files
opened 5000 files
opened 6000 files
opened 7000 files
opened 8000 files
opened 9000 files
opened 10000 files
opened 11000 files
opened 12000 files
opened 13000 files
opened 14000 files
opened 15000 files
opened 16000 files
opened 17000 files
opened 18000 files
opened 19000 files
opened 20000 files
opened 21000 files
opened 22000 files
opened 23000 files
opened 24000 files
opened 25000 files
opened 26000 files
opened 27000 files
opened 28000 files
opened 29000 files
opened 30000 files
opened 31000 files
opened 32000 files
Traceback (most recent call last):
  File "/home/autobahn/fdtest.py", line 10, in <module>
IOError: [Errno 24] Too many open files: '/tmp/test_32765'
^C
[autobahn at autobahnhub ~/AutobahnHub/service/autobahnhub/script]$





More information about the Twisted-Python mailing list