[Twisted-Python] Some beginner questions about "twisted.names.client" and ".tac" environment

Jesus Cea jcea at argo.es
Sat Dec 17 17:14:10 EST 2005


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Twisted 2.1, twisted.named 0.2, here.

I'm taking my first steps with Twisted (documentation -inexistence-
nightmare :-), and my first project will be a bulk mailer as the backend
of my mailing list system.

The application would take the message and the subscriber list and a)
resolve the MX for the domains and b) connect to the MX's and send the
message, trying to minimice traffic sending a single envelope for
several recipients sharing the domain or the MX's.

I'm doing currently the DNS stuff. The result are promising, resolving
about 200 domains per second in a 1.4GHz P4, so my biggest mailing list
(about 31500 unique domains, múltiple subscribers per domain) is
"resolved" in less than three minutes.

Nice so far. The demo code (2Kbytes) is the following (if I'm violating
the rules posting this code, please tell me):

=====
# File "dns.tac"

from twisted.application import service

application = service.Application("DNS test")

import time
t=time.time()

class resolucion(object) :
  def __init__(self,dominio) :
    from twisted.names import client
    d = client.lookupMailExchange(dominio,timeout=(60,))
    d.addCallbacks(self._cbMailExchange, self._ebMailExchange)
    self.dominio=dominio

  def _cbMailExchange(self,results):
    # Callback for MX query
    global aun_pendientes
    aun_pendientes-=1
    if not aun_pendientes :
      print "OK",time.time()-t
      return
      from twisted.internet import reactor
      reactor.stop()
      return
    if not len(pendientes) :
      return

    resolucion(pendientes.pop())
    from twisted.names.dns import QUERY_TYPES
    for i in results[0] :
      n=i.payload.name
      tipo=QUERY_TYPES[i.payload.TYPE]
      if tipo=="MX" :
        return
        p=i.payload.preference
        print n,p,
        for j in results[2] :
          if n==j.name :
            print j.payload.dottedQuad(),"(%d)" %j.ttl
            break
        else :
          print "???"
      elif tipo=="CNAME" :
        redirigidos.append((self.dominio,i.payload.name))

  def _ebMailExchange(self,failure):
    # Error callback for MX query
    global aun_pendientes
    aun_pendientes-=1
    if not aun_pendientes :
      print "ERROR",time.time()-t
      return
      from twisted.internet import reactor
      reactor.stop()
      return
    if not len(pendientes) :
      return

    resolucion(pendientes.pop())
    print "XXX",self.dominio
    print 'Lookup failed:'
    failure.printTraceback()

pendientes=[]
redirigidos=[]

f=open("domain_list")
for i in f :
  pendientes.append(i)

aun_pendientes=len(pendientes)

concurrencia=1000

for i in pendientes[:concurrencia] :
  resolucion(i)

from twisted.names import client
client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)]
# client.theResolver.resolvers=[client.theResolver.resolvers[-1]]

pendientes=pendientes[concurrencia:]

=====

I launch the code as "twistd -ny dns.tac".

The demo does 1000 resolutions in parallel. If you experiment with the
code, reduce the value.

Questions:

1. I get a warning: "[Uninitialized]
/usr/local/lib/python2.4/site-packages/twisted/names/dns.py:1227:
exceptions.DeprecationWarning: Deferred.setTimeout is deprecated.  Look
for timeout support specific to the API you are using instead."

 I'm using, the native "twisted.names" timeout API, as far as I know...

2. By default "twisted.names.client" uses the "/etc/resolv.conf" file to
know which nameservers to use. I, nevertheless, want to use a particular
nameserver, so:

 2.1. I couldn't to find an appropiate API. I had to do a "hack",
reading the "twisted.names" core to know implementation details:
"client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)]"

 2.2. The previous "hack" is only effective for future
"twisted.names.client" instances. The previous ones use the
"/etc/resolv.conf" entries. Putting the "hack" code before any instance
creation doesn't work.

 2.3. While reading the framework code, I saw that "client" uses a
resolver chain: host, cache, network. But the cache is initially clear
(of course) and NEVER ever gets populated, so we are not using it but
checking missing entries eats CPU: 155 seconds for the unchanged code,
125 seconds if I drop the host and cache resolvers.

 A caching client would be very nice, if the client is long running (my
original idea).

 2.4. The resolution failure code is only called if the resolution
timeouts. But if the domain doesn't exists, the code called is the
"success" one, with a "nil" answer. So we can't diferenciate between
inexistant domains and inexistant RRs.

3. How can I stop this ".tac"?. If I do "reactor.stop()", I get an
infinite error, repeated forever:

=====

[twisted.names.dns.DNSDatagramProtocol (UDP)] Traceback (most recent
call last):
          File
"/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43,
in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File
"/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line
59, in callWithContext
            return self.currentContext().callWithContext(ctx, func,
*args, **kw)
          File
"/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line
37, in callWithContext
            return func(*args,**kw)
          File
"/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py",
line 139, in _doReadOrWrite
            why = getattr(selectable, method)()
        --- <exception caught here> ---
          File
"/usr/local/lib/python2.4/site-packages/twisted/internet/udp.py", line
113, in doRead
            data, addr = self.socket.recvfrom(self.maxPacketSize)
        exceptions.AttributeError: 'Port' object has no attribute 'socket'

=====

I must kill -9 the "twistd" process.

Thank you for your time and attention. Help greatly appreciated :-)

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at argo.es http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
                                      _/_/    _/_/          _/_/_/_/_/
PGP Key Available at KeyServ   _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBQ6SNsplgi5GaxT1NAQLMQwP/czYFLQ6+olTCvM0jdmMlaBgwxHsHdvxT
/2mhWqtyhIf1Kdh6FioFQq13xqCfZxFIkwuUwTlG+ZmkSYK1iWZEmaS0CGa5YmuA
d7miIFfL9Tfa3OLyV1nvqdCR3YtzH/ws9UuJ2DGnACRI++Of6gBVwGlhFDa7S57o
wZcsYWAS6Sk=
=/Pks
-----END PGP SIGNATURE-----




More information about the Twisted-Python mailing list