[Twisted-Python] Some beginner questions about "twisted.names.client" and ".tac" environment
Jesus Cea
jcea at argo.es
Sat Dec 17 17:14:10 EST 2005
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Twisted 2.1, twisted.named 0.2, here.
I'm taking my first steps with Twisted (documentation -inexistence-
nightmare :-), and my first project will be a bulk mailer as the backend
of my mailing list system.
The application would take the message and the subscriber list and a)
resolve the MX for the domains and b) connect to the MX's and send the
message, trying to minimice traffic sending a single envelope for
several recipients sharing the domain or the MX's.
I'm doing currently the DNS stuff. The result are promising, resolving
about 200 domains per second in a 1.4GHz P4, so my biggest mailing list
(about 31500 unique domains, múltiple subscribers per domain) is
"resolved" in less than three minutes.
Nice so far. The demo code (2Kbytes) is the following (if I'm violating
the rules posting this code, please tell me):
=====
# File "dns.tac"
from twisted.application import service
application = service.Application("DNS test")
import time
t=time.time()
class resolucion(object) :
def __init__(self,dominio) :
from twisted.names import client
d = client.lookupMailExchange(dominio,timeout=(60,))
d.addCallbacks(self._cbMailExchange, self._ebMailExchange)
self.dominio=dominio
def _cbMailExchange(self,results):
# Callback for MX query
global aun_pendientes
aun_pendientes-=1
if not aun_pendientes :
print "OK",time.time()-t
return
from twisted.internet import reactor
reactor.stop()
return
if not len(pendientes) :
return
resolucion(pendientes.pop())
from twisted.names.dns import QUERY_TYPES
for i in results[0] :
n=i.payload.name
tipo=QUERY_TYPES[i.payload.TYPE]
if tipo=="MX" :
return
p=i.payload.preference
print n,p,
for j in results[2] :
if n==j.name :
print j.payload.dottedQuad(),"(%d)" %j.ttl
break
else :
print "???"
elif tipo=="CNAME" :
redirigidos.append((self.dominio,i.payload.name))
def _ebMailExchange(self,failure):
# Error callback for MX query
global aun_pendientes
aun_pendientes-=1
if not aun_pendientes :
print "ERROR",time.time()-t
return
from twisted.internet import reactor
reactor.stop()
return
if not len(pendientes) :
return
resolucion(pendientes.pop())
print "XXX",self.dominio
print 'Lookup failed:'
failure.printTraceback()
pendientes=[]
redirigidos=[]
f=open("domain_list")
for i in f :
pendientes.append(i)
aun_pendientes=len(pendientes)
concurrencia=1000
for i in pendientes[:concurrencia] :
resolucion(i)
from twisted.names import client
client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)]
# client.theResolver.resolvers=[client.theResolver.resolvers[-1]]
pendientes=pendientes[concurrencia:]
=====
I launch the code as "twistd -ny dns.tac".
The demo does 1000 resolutions in parallel. If you experiment with the
code, reduce the value.
Questions:
1. I get a warning: "[Uninitialized]
/usr/local/lib/python2.4/site-packages/twisted/names/dns.py:1227:
exceptions.DeprecationWarning: Deferred.setTimeout is deprecated. Look
for timeout support specific to the API you are using instead."
I'm using, the native "twisted.names" timeout API, as far as I know...
2. By default "twisted.names.client" uses the "/etc/resolv.conf" file to
know which nameservers to use. I, nevertheless, want to use a particular
nameserver, so:
2.1. I couldn't to find an appropiate API. I had to do a "hack",
reading the "twisted.names" core to know implementation details:
"client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)]"
2.2. The previous "hack" is only effective for future
"twisted.names.client" instances. The previous ones use the
"/etc/resolv.conf" entries. Putting the "hack" code before any instance
creation doesn't work.
2.3. While reading the framework code, I saw that "client" uses a
resolver chain: host, cache, network. But the cache is initially clear
(of course) and NEVER ever gets populated, so we are not using it but
checking missing entries eats CPU: 155 seconds for the unchanged code,
125 seconds if I drop the host and cache resolvers.
A caching client would be very nice, if the client is long running (my
original idea).
2.4. The resolution failure code is only called if the resolution
timeouts. But if the domain doesn't exists, the code called is the
"success" one, with a "nil" answer. So we can't diferenciate between
inexistant domains and inexistant RRs.
3. How can I stop this ".tac"?. If I do "reactor.stop()", I get an
infinite error, repeated forever:
=====
[twisted.names.dns.DNSDatagramProtocol (UDP)] Traceback (most recent
call last):
File
"/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43,
in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File
"/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line
59, in callWithContext
return self.currentContext().callWithContext(ctx, func,
*args, **kw)
File
"/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line
37, in callWithContext
return func(*args,**kw)
File
"/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py",
line 139, in _doReadOrWrite
why = getattr(selectable, method)()
--- <exception caught here> ---
File
"/usr/local/lib/python2.4/site-packages/twisted/internet/udp.py", line
113, in doRead
data, addr = self.socket.recvfrom(self.maxPacketSize)
exceptions.AttributeError: 'Port' object has no attribute 'socket'
=====
I must kill -9 the "twistd" process.
Thank you for your time and attention. Help greatly appreciated :-)
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jcea at argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/
_/_/ _/_/ _/_/_/_/_/
PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBQ6SNsplgi5GaxT1NAQLMQwP/czYFLQ6+olTCvM0jdmMlaBgwxHsHdvxT
/2mhWqtyhIf1Kdh6FioFQq13xqCfZxFIkwuUwTlG+ZmkSYK1iWZEmaS0CGa5YmuA
d7miIFfL9Tfa3OLyV1nvqdCR3YtzH/ws9UuJ2DGnACRI++Of6gBVwGlhFDa7S57o
wZcsYWAS6Sk=
=/Pks
-----END PGP SIGNATURE-----
More information about the Twisted-Python
mailing list