[Twisted-Python] RE: Modifying a web proxy to use SSL

Adams, Larry LAdams at doubleclick.com
Wed May 30 15:20:56 MDT 2007


Here is a little more detail on my problem.  I've been able to verify
that the connection is made to the https server and that data is
received.  In fact, I've overridden the ProxyRequest.handleResponseEnd()
method to save this data to a database.  I can see the raw html and
image data in the database, but nothing is returned to the browser.
Does anyone have any experience implementing their own proxy with SSL
that might be able to point me in the right direction?  Any assistance
would be greatly appreciated.
 
-larry
 
Here is my custom ProxyRequest class (i had to modify some stuff because
secure urls have the https stripped and :443 added to the uri for some
reason):
 
class SerpicoProxyRequest(proxy.ProxyRequest):
    #protocols = {'http': SerpicoProxyClientFactory}
    #LEA - added https
    protocols = {'http': SerpicoProxyClientFactory, 'https' :
SerpicoProxyClientFactory}
    ports = {'http' : 80, 'https' : 443}
 
    def __init__(self, *args):
        proxy.ProxyRequest.__init__(self, *args)
 
    def process(self):
        if (self.uri.find(':443')>0):
            self.uri = 'https://' + self.uri.replace(':443','')
            self.method = 'GET'
            parsed = urlparse.urlparse(self.uri)
            protocol = parsed[0]
            host = parsed[1]
            port = self.ports[protocol]
            self.setHost(host,port,1)
            if ':' in host:
                host, port = host.split(':')
                port = int(port)
 
            rest = urlparse.urlunparse(('','')+parsed[2:])
            if not rest:
                rest = rest+'/'
            class_ = self.protocols[protocol]
            headers = self.getAllHeaders().copy()
            if not headers.has_key('host'):
                headers['host'] = host
            self.content.seek(0,0)
            s = self.content.read()
            clientFactory = class_(self.method, rest, self.clientproto,
headers, s, self)
            clientContextFactory = ssl.ClientContextFactory()
            c = reactor.connectSSL(host, port, clientFactory,
clientContextFactory)
        else:
            proxy.ProxyRequest.process(self)
 
Here is my custom ProxyClient (the Page() object is what I'm using to
save the data to the database):
 
class SerpicoProxyClient(proxy.ProxyClient):
    def connectionMade(self):
        global CURRENTJOBID
        global CURRENTCHILDID
        log.msg("initializing connection")
        self.page = Page()
        if not self.father.uri == "%s/command/get_agent_task/%s" %
(command_site,machine_name):
            self.page.instance = machine_name
            self.page.job_id = int(CURRENTJOBID or 0)
            self.page.task_id = int(CURRENTCHILDID or 0)
            # self.father refers to the http.Request object
            self.page.request_uri = self.father.uri
            self.page.save()
            log.msg("MACHINE NAME IS %s" % machine_name)
            log.msg("PROXYREQUEST %s" % self.father.uri)
            log.msg("TRANSPORT %s" % self.transport)
            log.msg("ISSECURE %s" % self.father.isSecure())
        proxy.ProxyClient.connectionMade(self)
 
    def handleHeader(self, key, value):
        proxy.ProxyClient.handleHeader(self, key, value)
#        log.msg("%s : %s added" % (key, value))
        if not self.father.uri == "%s/command/get_agent_task/%s" %
(command_site,machine_name):
            self.page.header_set.create(key=key, value=value)
 
    def handleResponsePart(self, data):
        if not self.father.uri == "%s/command/get_agent_task/%s" %
(command_site,machine_name):
            self.page.data = data
        proxy.ProxyClient.handleResponsePart(self, data)
 
    def handleResponseEnd(self):
        if not self.father.uri == "%s/command/get_agent_task/%s" %
(command_site,machine_name):
            self.page.save()
        log.msg('ending response with my
data::%s\n\nfatherdata::%s\n\nmytransport::%s' % (self.page.data,
self.father.c
hannel.transport, self.transport))
        self.transport.loseConnection()
        self.father.channel.transport.loseConnection()
 

________________________________

From: Adams, Larry 
Sent: Tuesday, May 29, 2007 3:42 PM
To: twisted-python at twistedmatrix.com
Subject: Modifying a web proxy to use SSL



I'm trying to use twisted to proxy all web requests for a set of PCs
(virtual machines actually). I've created my own ProxyRequest class to
override the process() method so that it will use SSL if the URL is
secure. It makes the connection and retrieves the data successfully, but
the browser still doesn't display the HTML. I must be missing some other
class/method that I have to override, but I'm at a loss as to which one
at the moment.

Below is some logging from my application for the same page called via
http and https respectively. Any help or pointers to working examples
would be greatly appreciated!

Thanks,
Larry Adams 
2007/05/29 15:10 -0500 [SerpicoProxy,11,172.18.36.22] Starting factory
<__main__.SerpicoProxyClientFactory instance at 0xb71d8cec>

2007/05/29 15:10 -0500 [Uninitialized] building protocol for addr:
IPv4Address(TCP, 'www.connectcommerce.com
<file://www.connectcommerce.com> ', 80)
2007/05/29 15:10 -0500 [Uninitialized] initializing connection
2007/05/29 15:10 -0500 [Uninitialized] MACHINE NAME IS surfsidekick
2007/05/29 15:10 -0500 [Uninitialized] PROXYREQUEST
http://www.connectcommerce.com/ <http://www.connectcommerce.com/> 
2007/05/29 15:10 -0500 [Uninitialized] TRANSPORT <<class
'twisted.internet.tcp.Client'> to ('www.connectcommerce.com
<file://www.connectcommerce.com> ', 80) at b71d8ccc>

2007/05/29 15:10 -0500 [Uninitialized] ISSECURE False
2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Date : Tue, 29 May
2007 20:11:27 GMT added
2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Server :
Apache/1.3.33 (Unix) (Gentoo/Linux) added
2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Location :
http://www.connectcommerce.com/global/login.html
<http://www.connectcommerce.com/global/login.html>  added
2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Connection : close
added
2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Content-Type :
text/html; charset=iso-8859-1 added
2007/05/29 15:10 -0500 [SerpicoProxyClient,client] trying to lose
connection for transport: <<class 'twisted.internet.tcp.Client'> to
('www.connectcommerce.com <file://www.connectcommerce.com> ', 80) at
b71d8ccc>

2007/05/29 15:10 -0500 [SerpicoProxyClient,client] parent transport :
<SerpicoProxy #11 on 74007>
2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Stopping factory
<__main__.SerpicoProxyClientFactory instance at 0xb71d8cec>

2007/05/29 15:11 -0500 [SerpicoProxy,18,172.18.36.22] Starting factory
<__main__.SerpicoProxyClientFactory instance at 0xb71d8d0c>

2007/05/29 15:11 -0500 [Uninitialized] building protocol for addr:
IPv4Address(TCP, 'www.connectcommerce.com
<file://www.connectcommerce.com> ', 443)
2007/05/29 15:11 -0500 [Uninitialized] initializing connection
2007/05/29 15:11 -0500 [Uninitialized] MACHINE NAME IS surfsidekick
2007/05/29 15:11 -0500 [Uninitialized] PROXYREQUEST
https://www.connectcommerce.com <https://www.connectcommerce.com> 
2007/05/29 15:11 -0500 [Uninitialized] TRANSPORT <<class
'twisted.internet.tcp.TLSConnection'> to ('www.connectcommerce.com
<file://www.connectcommerce.com> ', 443) at b71bd86c>

2007/05/29 15:11 -0500 [Uninitialized] ISSECURE True
2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Date : Tue, 29 May
2007 20:12:00 GMT added
2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Server :
Apache/1.3.33 (Unix) (Gentoo/Linux) added
2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Location :
http://www.connectcommerce.com/global/login.html
<http://www.connectcommerce.com/global/login.html>  added
2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Connection : close
added
2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Content-Type :
text/html; charset=iso-8859-1 added
2007/05/29 15:11 -0500 [SerpicoProxyClient,client] trying to lose
connection for transport: <<class 'twisted.internet.tcp.TLSConnection'>
to ('www.connectcommerce.com <file://www.connectcommerce.com> ', 443) at
b71bd86c>

2007/05/29 15:11 -0500 [SerpicoProxyClient,client] parent transport :
<SerpicoProxy #18 on 74007>
2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Stopping factory
<__main__.SerpicoProxyClientFactory instance at 0xb71d8d0c>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20070530/890d0334/attachment.html>


More information about the Twisted-Python mailing list