Too many open (leaking?) File Descriptors and "cannot import name SHA"

[cb]

Hello

my name is Christian Bahls i am an anti-censorship activist from germany

we have this new law on internet-censorhip that would oblige providers to filter dns

so i set out and implement a dns-(json+http)-dns-tunnel

My Resolver looks like this:

resolvers=[] cacheResolver = cache.CacheResolver(verbose=True) resolvers.append(cacheResolver) hostResolver = hosts.Resolver(file='named.root.txt') #clientResolver = client.Resolver(servers=[('139.30.8.7',53)]) rootResolver = root.bootstrap(hostResolver) resolvers.append(rootResolver) #resolvers.append(hostResolver) #resolvers.append(clientResolver) theResolver = resolve.ResolverChain(resolvers)

it gets called like this:

def render_GET(self,request):

DNSLogger.info("REQUEST: %s" % (request)) d=theResolver.typeToMethod[self.typeKey](self.query) d.addCallback(self.returnResults,request) d.addErrback(self.returnError,request) #request.notifyFinish().addErrback(self.cancelRequest, d, request) return NOT_DONE_YET

where typeKey would be '1' for type 'IN'

Under heavy Pressure .. I get:

Unhandled error in Deferred: Traceback (most recent call last):

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 238, in callback

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 307, in _startRunCallbacks

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 323, in _runCallbacks

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 625, in gotResult

--- <exception caught here> ---

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 600, in _deferGenerator

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 120, in discoverAuthority

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 67, in lookupNameservers

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 31, in retry

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1730, in query

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1577, in pickID

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 44, in randomSource

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/python/randbytes.py", line 113, in secureRandom

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/python/randbytes.py", line 92, in _cryptoRandom

File "/usr/lib/python2.5/site-packages/Crypto/Util/randpool.py", line 57, in init

exceptions.ImportError: cannot import name SHA

####### AND AFTERWARDS ######### Unhandled error in Deferred: Traceback (most recent call last):

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 600, in _deferGenerator

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 120, in discoverAuthority

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 67, in lookupNameservers

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 31, in retry

--- <exception caught here> ---

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1725, in query

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1666, in startListening

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/posixbase.py", line 248, in listenUDP

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/udp.py", line 83, in startListening

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/udp.py", line 91, in _bindSocket

twisted.internet.error.CannotListenError: Couldn't listen on any:0: (24, 'Too many open files').

I saw a similar Bug arround here .. .. but it was supposed to be fixed?

.. as i have only the Deferred to play around with

i am not so sure how to get rid of the open UDP ports

yours Christian Bahls

[cb]

Well .. i had a short look into twisted.names.root

there are two sources for DNSDatagramProtocol leaks i think

def lookupNameservers(host, atServer, p=None):

# print 'Nameserver lookup for', host, 'at', atServer, 'with', p if p is None:

p = dns.DNSDatagramProtocol(_DummyController()) p.noisy = False

return retry(

(1, 3, 11, 45), # Timeouts p, # Protocol instance (atServer, dns.PORT), # Server to query [dns.Query(host, dns.NS, dns.IN)] # Question to ask

)

def lookupAddress(host, atServer, p=None):

# print 'Address lookup for', host, 'at', atServer, 'with', p if p is None:

p = dns.DNSDatagramProtocol(_DummyController()) p.noisy = False

return retry(

(1, 3, 11, 45), # Timeouts p, # Protocol instance (atServer, dns.PORT), # Server to query [dns.Query(host, dns.A, dns.IN)] # Question to ask

)

lookupNamerservers() is being called from

def discoverAuthority(host, roots, cache=None, p=None):

msg = defer.waitForDeferred(lookupNameservers(soFar, authority, p))

with p being 'None'

discoverAuthority() is being called on every _lookup()

def _lookup(self, name, cls, type, timeout):

d = discoverAuthority(name, self.hints

).addCallback(self.discoveredAuthority, name, cls, type, timeout )

return d

additionally discoveredAuthority() initializes a Resolver for every request:

def discoveredAuthority(self, auth, name, cls, type, timeout):

from twisted.names import client q = dns.Query(name, type, cls) r = client.Resolver(servers=[(auth, dns.PORT)]) d = r.queryUDP([q], timeout) d.addCallback(r.filterAnswers) return d

perhaps it could be done as in lookupAddress()

still this does not solve the leaking of protocols what to do about that?

yours

Christian Bahls

[cb]

snippets from twisted/names/root.py

[cb]

How the service is run .. removed all the specific stuff ..

[exarkun]

I tried running howdoicallit.py, but I get this on stdout:

Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: m.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: l.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: k.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: j.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: i.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: h.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: g.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: f.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: e.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: d.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: c.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: b.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: a.root-servers.net

And I don't see how I'm supposed to interact with the web server. Can you provide a self-contained example, or instructions on how to use this example?

[cb]

named.root for bootstrapping the root resolver

[cb]

of course .. you will need named.root.txt for bootstrapping the rootResolver with the hostResolver

you interact by getting * http://127.0.0.1:8881/IN/A/whatever * http://127.0.0.1:8881/IN/PTR/4.3.2.1.in-addr.arpa * http://127.0.0.1:8881/IN/ALL_RECORDS/whatever

but the Problem isn't with the Webserver ..

.. the Problem is that the rootResolver randomizes it's port by generating dns.DNSDatagramProtocol()s all the Time and never goes arround to actually get rid of them

so you can get the same Effect by just calling a rootResolvers _lookup() routine repeatedly

i will attach an example .. .. it will need the named.root.txt as a hint for bootstrapping the rootResolver

(ah .. and catching KeyboardInterrupt does not work as expected .. sorry for the bad failure handling)

Christia

[cb]

This file calles rootResolver repeatedly .. and shows the leakage even faster

[exarkun]

Ah, okay. This is a duplicate of #970.

[cb]

I am not using Windows .. .. this is not the same Bug

[exarkun]

The mention of Windows on #970 is a mistake, it applies to all platforms.

[cb]

No .. the solution in #970 does not help me .. as it uses the ThreadedResolver() which in turn calls socket.gethostbyname

gethostbyname will only end up asking the root.Resolver i am just trying to instantiate

the root.Resolver should be more careful with the resources being used

see: http://twistedmatrix.com/trac/ticket/4141#comment:4

if this is an error that will never be resolved than just tell me right away .. than i will stop bothering you with my requests ..

Christian

[exarkun]

#970 is still open, it has no solution yet. The ThreadedResolver change is just a comment someone put on the ticket. It wasn't an accepted solution, so the ticket is still open.

You're right that root.Resolver isn't cleaning up its resources. Fixing that problem is just what I think the solution to #970 should be.

[cb]

to me it looks as if the asynchronous Eventhandling has a design-bug

it boils down to this:

• how and when to throw away DNSDatagramProtocol-Instances (which would mean to somehow keep track of their usage)

yours

Christian

[thijs]

#970 mentions this ticket as duplicate?

[exarkun]

Yes. The fix for #970 will take care of this too.