<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>Modifying a web proxy to use SSL</TITLE>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3059" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2>Here is a little more detail on my problem. I've been
able to verify that the connection is made to the https server and that data is
received. In fact, I've overridden the </FONT></SPAN><SPAN
class=281121621-30052007><FONT face=Arial color=#0000ff
size=2>ProxyRequest.handleResponseEnd() method to save this data to a
database. I can see the raw html and image data in the database, but
nothing is returned to the browser. Does anyone have any experience
implementing their own proxy with SSL that might be able to point me in the
right direction? Any assistance would be greatly
appreciated.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2>-larry</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2>Here is my custom ProxyRequest class (i had to modify some
stuff because secure urls have the https stripped and :443 added to the uri for
some reason):</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2>class
SerpicoProxyRequest(proxy.ProxyRequest):<BR> #protocols =
{'http': SerpicoProxyClientFactory}<BR> #LEA - added
https<BR> protocols = {'http': SerpicoProxyClientFactory,
'https' : SerpicoProxyClientFactory}<BR> ports = {'http' : 80,
'https' : 443}</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2> def __init__(self,
*args):<BR>
proxy.ProxyRequest.__init__(self, *args)</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2> def
process(self):<BR> if
(self.uri.find(':443')>0):<BR>
self.uri = 'https://' +
self.uri.replace(':443','')<BR>
self.method =
'GET'<BR>
parsed =
urlparse.urlparse(self.uri)<BR>
protocol =
parsed[0]<BR>
host =
parsed[1]<BR>
port =
self.ports[protocol]<BR>
self.setHost(host,port,1)<BR>
if ':' in
host:<BR>
host, port =
host.split(':')<BR>
port = int(port)</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff
size=2> rest =
urlparse.urlunparse(('','')+parsed[2:])<BR>
if not
rest:<BR>
rest =
rest+'/'<BR>
class_ =
self.protocols[protocol]<BR>
headers =
self.getAllHeaders().copy()<BR>
if not
headers.has_key('host'):<BR>
headers['host'] =
host<BR>
self.content.seek(0,0)<BR>
s =
self.content.read()<BR>
clientFactory = class_(self.method, rest, self.clientproto, headers, s,
self)<BR>
clientContextFactory =
ssl.ClientContextFactory()<BR>
c = reactor.connectSSL(host, port, clientFactory,
clientContextFactory)<BR>
else:<BR>
proxy.ProxyRequest.process(self)</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2>Here is my custom ProxyClient (the Page() object is what
I'm using to save the data to the database):</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2>class
SerpicoProxyClient(proxy.ProxyClient):<BR> def
connectionMade(self):<BR> global
CURRENTJOBID<BR> global
CURRENTCHILDID<BR>
log.msg("initializing connection")<BR>
self.page = Page()<BR> if not
self.father.uri == "%s/command/get_agent_task/%s" %
(command_site,machine_name):<BR>
self.page.instance =
machine_name<BR>
self.page.job_id = int(CURRENTJOBID or
0)<BR>
self.page.task_id = int(CURRENTCHILDID or
0)<BR> #
self.father refers to the http.Request
object<BR>
self.page.request_uri =
self.father.uri<BR>
self.page.save()<BR>
log.msg("MACHINE NAME IS %s" %
machine_name)<BR>
log.msg("PROXYREQUEST %s" %
self.father.uri)<BR>
log.msg("TRANSPORT %s" %
self.transport)<BR>
log.msg("ISSECURE %s" %
self.father.isSecure())<BR>
proxy.ProxyClient.connectionMade(self)</FONT></SPAN></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2> def handleHeader(self, key,
value):<BR>
proxy.ProxyClient.handleHeader(self, key,
value)<BR># log.msg("%s : %s added" %
(key, value))<BR> if not
self.father.uri == "%s/command/get_agent_task/%s" %
(command_site,machine_name):<BR>
self.page.header_set.create(key=key, value=value)</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2> def handleResponsePart(self,
data):<BR> if not self.father.uri ==
"%s/command/get_agent_task/%s" %
(command_site,machine_name):<BR>
self.page.data = data<BR>
proxy.ProxyClient.handleResponsePart(self, data)</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2> def
handleResponseEnd(self):<BR> if not
self.father.uri == "%s/command/get_agent_task/%s" %
(command_site,machine_name):<BR>
self.page.save()<BR> log.msg('ending
response with my data::%s\n\nfatherdata::%s\n\nmytransport::%s' %
(self.page.data, self.father.c<BR>hannel.transport,
self.transport))<BR>
self.transport.loseConnection()<BR>
self.father.channel.transport.loseConnection()</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=281121621-30052007><FONT face=Arial
color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Adams, Larry <BR><B>Sent:</B> Tuesday, May
29, 2007 3:42 PM<BR><B>To:</B>
twisted-python@twistedmatrix.com<BR><B>Subject:</B> Modifying a web proxy to use
SSL<BR></FONT><BR></DIV>
<DIV></DIV><!-- Converted from text/rtf format -->
<P><FONT face=Arial size=2>I'm trying to use twisted to proxy all web requests
for a set of PCs (virtual machines actually). I've created my own ProxyRequest
class to override the process() method so that it will use SSL if the URL is
secure. It makes the connection and retrieves the data successfully, but the
browser still doesn't display the HTML. I must be missing some other
class/method that I have to override, but I'm at a loss as to which one at the
moment.</FONT></P>
<P><FONT face=Arial size=2>Below is some logging from my application for the
same page called via http and https respectively. Any help or pointers to
working examples would be greatly appreciated!</FONT></P>
<P><FONT face=Arial size=2>Thanks,<BR>Larry Adams</FONT><FONT
face="Times New Roman"> </FONT><BR><FONT face=Arial size=2>2007/05/29 15:10
-0500 [SerpicoProxy,11,172.18.36.22] Starting factory
<__main__.SerpicoProxyClientFactory instance at 0xb71d8cec></FONT></P>
<P><FONT face=Arial size=2>2007/05/29 15:10 -0500 [Uninitialized] building
protocol for addr: IPv4Address(TCP, '</FONT><A
href="file://www.connectcommerce.com"><U></U><U></U><U><FONT face=Arial
color=#0000ff size=2>www.connectcommerce.com</FONT></U></A><FONT face=Arial
size=2>', 80)</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:10 -0500 [Uninitialized] initializing
connection</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:10 -0500 [Uninitialized] MACHINE NAME IS
surfsidekick</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:10 -0500 [Uninitialized] PROXYREQUEST </FONT><A
href="http://www.connectcommerce.com/"><U><FONT face=Arial color=#0000ff
size=2>http://www.connectcommerce.com/</FONT></U></A><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>2007/05/29 15:10 -0500
[Uninitialized] TRANSPORT <<class 'twisted.internet.tcp.Client'> to
('</FONT><A href="file://www.connectcommerce.com"><U></U><U></U><U><FONT
face=Arial color=#0000ff size=2>www.connectcommerce.com</FONT></U></A><FONT
face=Arial size=2>', 80) at b71d8ccc></FONT></P>
<P><FONT face=Arial size=2>2007/05/29 15:10 -0500 [Uninitialized] ISSECURE
False</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Date : Tue, 29 May
2007 20:11:27 GMT added</FONT><FONT face="Times New Roman"><BR></FONT><FONT
face=Arial size=2>2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Server :
Apache/1.3.33 (Unix) (Gentoo/Linux) added</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>2007/05/29 15:10 -0500
[SerpicoProxyClient,client] Location : </FONT><A
href="http://www.connectcommerce.com/global/login.html"><U><FONT face=Arial
color=#0000ff
size=2>http://www.connectcommerce.com/global/login.html</FONT></U></A><FONT
face=Arial size=2> added</FONT><FONT face="Times New Roman"><BR></FONT><FONT
face=Arial size=2>2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Connection
: close added</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:10 -0500 [SerpicoProxyClient,client] Content-Type :
text/html; charset=iso-8859-1 added</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>2007/05/29 15:10 -0500
[SerpicoProxyClient,client] trying to lose connection for transport:
<<class 'twisted.internet.tcp.Client'> to ('</FONT><A
href="file://www.connectcommerce.com"><U></U><U></U><U><FONT face=Arial
color=#0000ff size=2>www.connectcommerce.com</FONT></U></A><FONT face=Arial
size=2>', 80) at b71d8ccc></FONT></P>
<P><FONT face=Arial size=2>2007/05/29 15:10 -0500 [SerpicoProxyClient,client]
parent transport : <SerpicoProxy #11 on 74007></FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>2007/05/29 15:10 -0500
[SerpicoProxyClient,client] Stopping factory
<__main__.SerpicoProxyClientFactory instance at 0xb71d8cec></FONT></P>
<P><FONT face=Arial size=2>2007/05/29 15:11 -0500 [SerpicoProxy,18,172.18.36.22]
Starting factory <__main__.SerpicoProxyClientFactory instance at
0xb71d8d0c></FONT></P>
<P><FONT face=Arial size=2>2007/05/29 15:11 -0500 [Uninitialized] building
protocol for addr: IPv4Address(TCP, '</FONT><A
href="file://www.connectcommerce.com"><U></U><U></U><U><FONT face=Arial
color=#0000ff size=2>www.connectcommerce.com</FONT></U></A><FONT face=Arial
size=2>', 443)</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:11 -0500 [Uninitialized] initializing
connection</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:11 -0500 [Uninitialized] MACHINE NAME IS
surfsidekick</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:11 -0500 [Uninitialized] PROXYREQUEST </FONT><A
href="https://www.connectcommerce.com"><U><FONT face=Arial color=#0000ff
size=2>https://www.connectcommerce.com</FONT></U></A><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>2007/05/29 15:11 -0500
[Uninitialized] TRANSPORT <<class 'twisted.internet.tcp.TLSConnection'>
to ('</FONT><A href="file://www.connectcommerce.com"><U></U><U></U><U><FONT
face=Arial color=#0000ff size=2>www.connectcommerce.com</FONT></U></A><FONT
face=Arial size=2>', 443) at b71bd86c></FONT></P>
<P><FONT face=Arial size=2>2007/05/29 15:11 -0500 [Uninitialized] ISSECURE
True</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Date : Tue, 29 May
2007 20:12:00 GMT added</FONT><FONT face="Times New Roman"><BR></FONT><FONT
face=Arial size=2>2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Server :
Apache/1.3.33 (Unix) (Gentoo/Linux) added</FONT><FONT
face="Times New Roman"><BR><B></B></FONT><B><FONT face=Arial size=2>2007/05/29
15:11 -0500 [SerpicoProxyClient,client] Location :</FONT></B> <A
href="http://www.connectcommerce.com/global/login.html"><U><FONT face=Arial
color=#0000ff
size=2>http://www.connectcommerce.com/global/login.html</FONT></U></A><B><FONT
face=Arial size=2> added</FONT></B><FONT face="Times New Roman"><BR></FONT><FONT
face=Arial size=2>2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Connection
: close added</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>2007/05/29 15:11 -0500 [SerpicoProxyClient,client] Content-Type :
text/html; charset=iso-8859-1 added</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>2007/05/29 15:11 -0500
[SerpicoProxyClient,client] trying to lose connection for transport:
<<class 'twisted.internet.tcp.TLSConnection'> to ('</FONT><A
href="file://www.connectcommerce.com"><U></U><U></U><U><FONT face=Arial
color=#0000ff size=2>www.connectcommerce.com</FONT></U></A><FONT face=Arial
size=2>', 443) at b71bd86c></FONT></P>
<P><FONT face=Arial size=2>2007/05/29 15:11 -0500 [SerpicoProxyClient,client]
parent transport : <SerpicoProxy #18 on 74007></FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>2007/05/29 15:11 -0500
[SerpicoProxyClient,client] Stopping factory
<__main__.SerpicoProxyClientFactory instance at
0xb71d8d0c></FONT></P></BODY></HTML>