[Twisted-web] Noob question about POST

Donald Steiny steiny at steiny.com
Sat Dec 31 14:22:32 EST 2011


Hi,

I have just been learning twisted and been Googling for an example of 
something that is simple to do with pycurl or even straight python but I 
can't figure out for the life of me how to do with twisted.   I want to 
be able to do a POST and debug a login to a system.  Ultimately, what I 
am trying to do (and twisted seems to be the best choice for this) is to 
make the login process automatic for the user so after he or she is 
logged in he or she will interact with the remote system as if he  or 
she had logged in his or herself.  The proxy class seems perfect for the 
second part and getPage should allow me to do the first, but I don't 
seem to understand something.  What I have done is to set up a simple 
login/password page on my own server with login fred and password 1234.  
I am using Python 2.6 on Linux 2.6. I will describe what I expect to 
happen, then what I did, then what happened that I did not expect and if 
anyone could set me straight I would be eternally grateful AND because 
of the good Karma you will find parking spaces easily all during 2012.

If I telnet to localhost port 80 and copy and paste the following:

=== BEGINNING OF WORKING INPUT TO SERVER  ===
POST /locked/checklogin.php HTTP/1.1
Host: www.mysite.com
User-Agent: Mozilla/4.0
Content-Length: 31
Content-Type: application/x-www-form-urlencoded

myusername=john&mypassword=1234
===== END OF WORKING INPUT TO SERVER ====

I get just what I would expect.  On successful login it goes to the page 
"login_success.php"and I print out the _POST array to make sure the 
variables are getting passed:

==== BEGINNING OF CORRECT OUTPUT FROM SERVER ===
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
POST /locked/checklogin.php HTTP/1.1
Host: www.mysite.com
User-Agent: Mozilla/4.0
Content-Length: 31
Content-Type: application/x-www-form-urlencoded

myusername=john&mypassword=1234
HTTP/1.1 302 Found
Date: Sat, 31 Dec 2011 18:42:28 GMT
Server: Apache/2.2.16 (Debian)
X-Powered-By: PHP/5.3.3-7+squeeze3
Set-Cookie: PHPSESSID=tueqdd71kha040d18d1qabg424; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, 
pre-check=0
Pragma: no-cache
Location: login_success.php
Vary: Accept-Encoding
Content-Length: 57
Content-Type: text/html

IN CHECKLOGIN<BR>
myusername: john, mypassword: 1234<BR>
Connection closed by foreign host.
=== END OF CORRECT OUTPUT FROM SERVER ===

I took a getPage example that is in the documentation:

=== START CODE ===
from pprint import pformat

from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers

class BeginningPrinter(Protocol):
     def __init__(self, finished):
         self.finished = finished
         self.remaining = 1024 * 10

     def dataReceived(self, bytes):
         if self.remaining:
             display = bytes[:self.remaining]
             print 'Some data received:'
             print display
             self.remaining -= len(display)

     def connectionLost(self, reason):
         print 'Finished receiving body:', reason.getErrorMessage()
         self.finished.callback(None)

agent = Agent(reactor)
d = agent.request(
     'GET',
     'http://localhost/locked/checklogin.php',
     Headers({'User-Agent': ['Twisted Web Client Example']}),
     None)

def cbRequest(response):
     print 'Response version:', response.version
     print 'Response code:', response.code
     print 'Response phrase:', response.phrase
     print 'Response headers:'
     print pformat(list(response.headers.getAllRawHeaders()))
     finished = Deferred()
     response.deliverBody(BeginningPrinter(finished))
     return finished
d.addCallback(cbRequest)

def cbShutdown(ignored):
     reactor.stop()
d.addBoth(cbShutdown)

reactor.run()

==== END CODE ===

The output was just what I expected:

=== OUTPUT FROM RUN OF CODE ==

Response version: ('HTTP', 1, 1)
Response code: 200
Response phrase: OK
Response headers:
[('Date', ['Sat, 31 Dec 2011 18:55:17 GMT']),
  ('Content-Type', ['text/html']),
  ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']),
  ('Vary', ['Accept-Encoding']),
  ('Server', ['Apache/2.2.16 (Debian)'])]
Some data received:
IN CHECKLOGIN<BR>
myusername: , mypassword: <BR>
Wrong Username or Password: ,
Finished receiving body: Response body fully received
shutdown called
===== END OF OUTPUT FROM RUN OF CODE --

I modified the example by changing the GET to a POST and including 
StringProducer and changing the None to a body and the header to 
multipart/form-data.  Here:

==== START OF CODE MODIFIED WITH POST ===

from pprint import pformat

from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
from twisted.web.iweb import IBodyProducer
from zope.interface import implements, Interface
from stringprod import StringProducer

class BeginningPrinter(Protocol):
     def __init__(self, finished):
         self.finished = finished
         self.remaining = 1024 * 10

     def dataReceived(self, bytes):
         if self.remaining:
             display = bytes[:self.remaining]
             print 'Some data received:'
             print display
             self.remaining -= len(display)

     def connectionLost(self, reason):
         print 'Finished receiving body:', reason.getErrorMessage()
         self.finished.callback(None)

creds = "myusername=john&mypassword=1234\r\n"

body = StringProducer(creds)
agent = Agent(reactor)
d = agent.request(
     'POST',
     'http://localhost/locked/checklogin.php',
     Headers({'User-Agent': ['Mozilla 5.0'],
         'Content-Type': ['multipart/form-data; charset=utf-8']}),
     body)

def cbRequest(response):
     print 'Response version:', response.version
     print 'Response code:', response.code
     print 'Response phrase:', response.phrase
     print 'Response headers:'
     print pformat(list(response.headers.getAllRawHeaders()))
     finished = Deferred()
     response.deliverBody(BeginningPrinter(finished))
     return finished

d.addCallback(cbRequest)

def cbShutdown(ignored):
     print "shutdown called"
     reactor.stop()
d.addBoth(cbShutdown)

reactor.run()

=== END CODE MODIFIED WITH POST ===

When I run this, I do not get the output I expect, for some reason, it 
does not seem to be sending the body.

==== BEGINNING OF RUN OF MODIFIED CODE ====
Response version: ('HTTP', 1, 1)
Response code: 200
Response phrase: OK
Response headers:
[('Date', ['Sat, 31 Dec 2011 19:06:39 GMT']),
  ('Content-Type', ['text/html']),
  ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']),
  ('Vary', ['Accept-Encoding']),
  ('Server', ['Apache/2.2.16 (Debian)'])]
Some data received:
IN CHECKLOGIN<BR>
myusername: , mypassword: <BR>
Wrong Username or Password: ,
Finished receiving body: Response body fully received
shutdown called
  ==== END OF RUN OF MODIFIED CODE ===

What am I missing?  The body does not seem to be being send.   I would 
like to have a way to easily be able to debug logins to remote systems.  
This is NOT for spamming.  I am trying to develop a system for higher 
security where our users would log locally (VPN)  with their own 
favorite passwords and that would map to awful username and password 
that no one would ever use or remember and would be unguessable to log 
them into external cloud based systems.  In some cases there are 
libraries and APIs to help me, but I want a general solution.  Mechanic 
seems to do part of this and pycurl, but the proxy and networking part 
seem much better in twisted.  HELP!!!

Thanks,
Donald





More information about the Twisted-web mailing list