[Twisted-Python] data dispatch on massive connection counts

Donal McMullan donal.mcmullan at gmail.com
Mon Nov 14 10:15:20 EST 2011


What does resource.getrlimit(resource.RLIMIT_OFILE) return?

The Python docs say:

resource.RLIMIT_OFILE
 The BSD name for RLIMIT_NOFILE.

D



On 15 November 2011 03:57, Tobias Oberstein <tobias.oberstein at tavendo.de> wrote:
>> > 2) Too many files.
>> > Is there another limit specifically for files, and/or something tunable in
>> Python/Twisted?
>>
>> That's definitely not a limit on Python or Twisted. That's an OS limit.
>> Are you opening lots of files in addition to your sockets?
>
> Python open() will bail out at 32768 open files.
>
> This despite the fact that
>
> a) resource.getrlimit(resource.RLIMIT_NOFILE) = 200k
> b) I can accept 50k sockets
>
> Obviously, it's at least not Twisted related ..
>
> =========
>
> import resource
>
> max = resource.getrlimit(resource.RLIMIT_NOFILE)
> cnt = 0
> print "maximum FDs", max
>
> fds = []
>
> while cnt < max:
>   f = open("/tmp/test_%d" % cnt, "w")
>   f.write("test")
>   fds.append(f)
>   cnt += 1
>   if cnt % 1000 == 0:
>      print "opened %d files" % cnt
>
> print "ok, created %d files" % cnt
>
> =========
>
> maximum FDs (200000L, 200000L)
> opened 1000 files
> opened 2000 files
> opened 3000 files
> opened 4000 files
> opened 5000 files
> opened 6000 files
> opened 7000 files
> opened 8000 files
> opened 9000 files
> opened 10000 files
> opened 11000 files
> opened 12000 files
> opened 13000 files
> opened 14000 files
> opened 15000 files
> opened 16000 files
> opened 17000 files
> opened 18000 files
> opened 19000 files
> opened 20000 files
> opened 21000 files
> opened 22000 files
> opened 23000 files
> opened 24000 files
> opened 25000 files
> opened 26000 files
> opened 27000 files
> opened 28000 files
> opened 29000 files
> opened 30000 files
> opened 31000 files
> opened 32000 files
> Traceback (most recent call last):
>  File "/home/autobahn/fdtest.py", line 10, in <module>
> IOError: [Errno 24] Too many open files: '/tmp/test_32765'
> ^C
> [autobahn at autobahnhub ~/AutobahnHub/service/autobahnhub/script]$
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



More information about the Twisted-Python mailing list