Opened 5 years ago

Closed 5 years ago

#4141 defect closed duplicate (duplicate)

Too many open (leaking?) File Descriptors and "cannot import name SHA"

Reported by: cb Owned by: cb
Priority: normal Milestone:
Component: names Keywords:
Cc: thijs Branch:
Author: Launchpad Bug:

Description

Hello

my name is Christian Bahls
i am an anti-censorship activist from germany


we have this new law on internet-censorhip that
would oblige providers to filter dns

so i set out and implement a dns-(json+http)-dns-tunnel

My Resolver looks like this:

resolvers=[]
cacheResolver = cache.CacheResolver(verbose=True)
resolvers.append(cacheResolver)
hostResolver = hosts.Resolver(file='named.root.txt')
#clientResolver = client.Resolver(servers=[('139.30.8.7',53)])
rootResolver = root.bootstrap(hostResolver)
resolvers.append(rootResolver)
#resolvers.append(hostResolver)
#resolvers.append(clientResolver)
theResolver = resolve.ResolverChain(resolvers)

it gets called like this:

def render_GET(self,request):

DNSLogger.info("REQUEST: %s" % (request))
d=theResolver.typeToMethod[self.typeKey](self.query)
d.addCallback(self.returnResults,request)
d.addErrback(self.returnError,request)
#request.notifyFinish().addErrback(self.cancelRequest, d, request)
return NOT_DONE_YET

where typeKey would be '1' for type 'IN'

Under heavy Pressure .. I get:

Unhandled error in Deferred:
Traceback (most recent call last):

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 238, in callback


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 307, in _startRunCallbacks


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 323, in _runCallbacks


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 625, in gotResult


--- <exception caught here> ---

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 600, in _deferGenerator


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 120, in discoverAuthority


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 67, in lookupNameservers


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 31, in retry


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1730, in query


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1577, in pickID


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 44, in randomSource


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/python/randbytes.py", line 113, in secureRandom


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/python/randbytes.py", line 92, in _cryptoRandom


File "/usr/lib/python2.5/site-packages/Crypto/Util/randpool.py", line 57, in init


exceptions.ImportError: cannot import name SHA

####### AND AFTERWARDS #########
Unhandled error in Deferred:
Traceback (most recent call last):

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py", line 600, in _deferGenerator


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 120, in discoverAuthority


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 67, in lookupNameservers


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/root.py", line 31, in retry


--- <exception caught here> ---

File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1725, in query


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/names/dns.py", line 1666, in startListening


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/posixbase.py", line 248, in listenUDP


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/udp.py", line 83, in startListening


File "/usr/lib/python2.5/site-packages/Twisted-9.0.0-py2.5-linux-x86_64.egg/twisted/internet/udp.py", line 91, in _bindSocket


twisted.internet.error.CannotListenError: Couldn't listen on any:0: (24, 'Too many open files').

I saw a similar Bug arround here ..
.. but it was supposed to be fixed?

.. as i have only the Deferred to play around with

i am not so sure how to get rid of the open UDP ports

yours Christian Bahls

Attachments (4)

snippets.py (1.6 KB) - added by cb 5 years ago.
snippets from twisted/names/root.py
howdoicallit.py (5.1 KB) - added by cb 5 years ago.
How the service is run .. removed all the specific stuff ..
named.root.txt (663 bytes) - added by cb 5 years ago.
named.root for bootstrapping the root resolver
showtheleakage.py (1.2 KB) - added by cb 5 years ago.
This file calles rootResolver repeatedly .. and shows the leakage even faster

Download all attachments as: .zip

Change History (16)

comment:1 Changed 5 years ago by cb

Well .. i had a short look into twisted.names.root

there are two sources for DNSDatagramProtocol leaks i think

def lookupNameservers(host, atServer, p=None):

# print 'Nameserver lookup for', host, 'at', atServer, 'with', p
if p is None:

p = dns.DNSDatagramProtocol(_DummyController())
p.noisy = False

return retry(

(1, 3, 11, 45), # Timeouts
p, # Protocol instance
(atServer, dns.PORT), # Server to query
[dns.Query(host, dns.NS, dns.IN)] # Question to ask

)

def lookupAddress(host, atServer, p=None):

# print 'Address lookup for', host, 'at', atServer, 'with', p
if p is None:

p = dns.DNSDatagramProtocol(_DummyController())
p.noisy = False

return retry(

(1, 3, 11, 45), # Timeouts
p, # Protocol instance
(atServer, dns.PORT), # Server to query
[dns.Query(host, dns.A, dns.IN)] # Question to ask

)

lookupNamerservers() is being called from

def discoverAuthority(host, roots, cache=None, p=None):

msg = defer.waitForDeferred(lookupNameservers(soFar, authority, p))

with p being 'None'

discoverAuthority() is being called on every _lookup()

def _lookup(self, name, cls, type, timeout):

d = discoverAuthority(name, self.hints

).addCallback(self.discoveredAuthority, name, cls, type, timeout
)

return d

additionally discoveredAuthority() initializes a Resolver for every request:

def discoveredAuthority(self, auth, name, cls, type, timeout):

from twisted.names import client
q = dns.Query(name, type, cls)
r = client.Resolver(servers=[(auth, dns.PORT)])
d = r.queryUDP([q], timeout)
d.addCallback(r.filterAnswers)
return d

perhaps it could be done as in lookupAddress()

still this does not solve the leaking of protocols
what to do about that?

yours

Christian Bahls

Changed 5 years ago by cb

snippets from twisted/names/root.py

Changed 5 years ago by cb

How the service is run .. removed all the specific stuff ..

comment:2 Changed 5 years ago by cb

  • Priority changed from normal to high

comment:3 Changed 5 years ago by exarkun

  • Author Christian Bahls deleted
  • Owner changed from exarkun to cb
  • Priority changed from high to normal

I tried running howdoicallit.py, but I get this on stdout:

Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: m.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: l.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: k.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: j.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: i.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: h.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: g.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: f.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: e.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: d.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: c.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: b.root-servers.net
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: twisted.names.error.DomainError: a.root-servers.net

And I don't see how I'm supposed to interact with the web server. Can you provide a self-contained example, or instructions on how to use this example?

Changed 5 years ago by cb

named.root for bootstrapping the root resolver

comment:4 Changed 5 years ago by cb

of course .. you will need named.root.txt for bootstrapping the rootResolver
with the hostResolver

you interact by getting
*http://127.0.0.1:8881/IN/A/whatever
*http://127.0.0.1:8881/IN/PTR/4.3.2.1.in-addr.arpa
*http://127.0.0.1:8881/IN/ALL_RECORDS/whatever

but the Problem isn't with the Webserver ..

.. the Problem is that the rootResolver randomizes
it's port by generating dns.DNSDatagramProtocol()s all the Time
and never goes arround to actually get rid of them

so you can get the same Effect by just calling a
rootResolvers _lookup() routine repeatedly

i will attach an example ..
.. it will need the named.root.txt as a hint for bootstrapping the rootResolver

(ah .. and catching KeyboardInterrupt does not work as expected .. sorry for the bad failure handling)

Christia

Changed 5 years ago by cb

This file calles rootResolver repeatedly .. and shows the leakage even faster

comment:5 Changed 5 years ago by exarkun

  • Resolution set to duplicate
  • Status changed from new to closed

Ah, okay. This is a duplicate of #970.

comment:6 Changed 5 years ago by cb

  • Resolution duplicate deleted
  • Status changed from closed to reopened

I am not using Windows ..
.. this is not the same Bug

comment:7 Changed 5 years ago by exarkun

  • Resolution set to duplicate
  • Status changed from reopened to closed

The mention of Windows on #970 is a mistake, it applies to all platforms.

comment:8 Changed 5 years ago by cb

  • Resolution duplicate deleted
  • Status changed from closed to reopened

No .. the solution in #970 does not help me .. as it uses the ThreadedResolver() which in turn calls socket.gethostbyname

gethostbyname will only end up asking the root.Resolver i am just trying to instantiate

the root.Resolver should be more careful with the resources being used

see: http://twistedmatrix.com/trac/ticket/4141#comment:4

if this is an error that will never be resolved than just tell me right away .. than i will stop bothering you with my requests ..

Christian

comment:9 Changed 5 years ago by exarkun

#970 is still open, it has no solution yet. The ThreadedResolver change is just a comment someone put on the ticket. It wasn't an accepted solution, so the ticket is still open.

You're right that root.Resolver isn't cleaning up its resources. Fixing that problem is just what I think the solution to #970 should be.

comment:10 Changed 5 years ago by cb

to me it looks as if the asynchronous Eventhandling has a design-bug

it boils down to this:

  • how and when to throw away DNSDatagramProtocol-Instances (which would mean to somehow keep track of their usage)

yours

Christian

comment:11 Changed 5 years ago by thijs

  • Cc thijs added

#970 mentions this ticket as duplicate?

comment:12 Changed 5 years ago by exarkun

  • Resolution set to duplicate
  • Status changed from reopened to closed

Yes. The fix for #970 will take care of this too.

Note: See TracTickets for help on using tickets.