[Twisted-Python] getPage using ssl proxy

Konrads Smelkovs konrads at smelkovs.com
Thu Jul 30 15:15:31 EDT 2009


I am writing some scraper scripts and need to pass them through an
intercepting proxy. getPage does not support a proxy argument and this code
I found on internet won't work with SSL proxy (stalls indefinitely):

def getPage(url, contextFactory=None, *args, **kwargs):
    scheme, host, port, path = _parse(url)
    factory = HTTPClientFactory(url, *args, **kwargs)
    if 0: # use a proxy
        host, port = 'localhost', 8080
        factory.path = url
    if scheme == 'https':
        from twisted.internet import ssl
        if contextFactory is None:
            contextFactory = ssl.ClientContextFactory()
        reactor.connectSSL(host, port, factory, contextFactory)
        reactor.connectTCP(host, port, factory)
    return factory.deferred

Plain http proxying works. My guess is that there is an issue with
self-signed or otherwise invalid certificate the http proxy supplies. Any

Konrads Smelkovs
Applied IT sorcery.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090730/16e6f04a/attachment.htm 

More information about the Twisted-Python mailing list