[Twisted-Python] New TLS Code *Could* Cause Problems

[Reza Lotun]

Hi All,

I just wanted to fire a note about our experiences testing Twisted trunk
(after _newtls.py landed) on some servers in our cluster, on PyOpenSSL
0.12.

In our application (a twisted.web REST api endpoint), we make a lot of https
calls to a SimpleDB Amazon AWS endpoint (sdb.amazonaws.com). At some point
after a deployed test on some of our machines, we started getting our
internel DNS requests to the sdb endpoint throttled (resulting in failures
in most of our requests). Since every call to SimpleDB was made to the sdb
endpoint, each call would be resolved, bombarding amazon's dns servers with
requests. After diving in, I learned all Twisted connect* calls go through
socket.gethostbyname, which apparently does not do any DNS caching (much to
my surprise) unless something like nscd or a local caching nameserver is
running (we ended up using BIND).

Now, since we never ran a local caching nameserver before I wondered what
changed. Could it have been that using the _newtls code (with PyOpenSSL
0.12) removed some element of DNS caching that OpenSSL was performing behind
the scene? Is this possible? If so, then I just wanted to make sure everyone
was aware that if you make a lot of DNS calls in a Twisted server running
this code, you too could experience problems if you're not running a local
caching nameserver.

Thanks,
Reza

-- 
Reza Lotun
mobile: +44 (0)7521 310 763
email:  rlotun at gmail.com
work:   rlotun at twitter.com
@rlotun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110719/c30a0c52/attachment.htm