[Twisted-Python] Newbie - guidance required please

[Frank Millman]

Hi all,

I am treading a path that I think many have trodden before. I have a
requirement, I am pretty sure that Twisted will be able to solve it for me,
but at this stage I am overwhelmed by the options that Twisted seems to
offer, and I am worried that I will head off down the wrong path. With your
indulgence, I would like to describe what I am trying to accomplish, and get
some feedback as to which features of Twisted are best suited to assisting
me in achieving this.

I am developing a multi-user accounting/business system. I use a database
for the back end - PostgreSQL/psycopg on Linux, MS SQL Server/pywin32.odbc
on MSW. I am using wxPython for the front end, but eventually I will want a
web interface as well. I have developed a reasonably powerful middle layer
to handle the authentication and business logic. My problem is that at
present the middle layer runs on the client, each with its own connection to
the database. I am sure that there are a number of problems with this
architecture, but the one that concerns me particularly is that it is easy
for a user with a smattering of knowledge to modify the client program to
bypass my authentication and business logic, make a direct connection to the
database, and generate any SQL statements they want. This makes my approach
a non-starter.

The solution that I have in mind is to move all of my authentication and
business logic to a server program, which will be the only program that
makes a connection to the database. Each client will connect to my server
program, and I will have to devise some message formats for the passing of
information between the client and the server. I have a little experience
with multi-threaded socket programming, and I am fairly sure that I could
come up with a working solution using this technique, but having read the
advantages of the Twisted approach, I feel that this would be a better and
more scaleable solution.

Before going into detail, there is one issue I am not sure about, and that
is the question of 'maintaining state'. The user selects an option from a
menu - maintain a table, capture a transaction, run a report, perform an
enquiry, etc. Each option is represented by a class, which is contained in a
module of the same name. When an option is selected, I import the module,
and instantiate the class. Each option is associated with a number of
database tables, each with a number of columns. I have a generalised 'table
class', which is instantiated for each table opened, and a generalised
'column class', which is instantiated for each column. A user can have more
than one option running concurrently, which can result in dozens of table
objects and hundreds of column objects. I was fairly relaxed about this
running on each client, but I am not so sure about running them all on one
server with, say, 50 or more concurrent clients. Should I worry about the
number of active objects, or is it safe to assume that virtual memory will
handle it, on the assumption that most of them will be idle most of the
time?

As I understand it, I will use Twisted in the following roles -

1. in my server program, as an interface to DB-API for all database access.
2. in my server program, to handle requests from and pass information to
clients.
3. in my client program, to connect with and interact with my server
program.

Having read the documentation on twisted.enterprise.adbapi, it seems that
'1' should be fairly straightforward. I hope there are not too many hidden
complications.

I assume that '2' and '3' can be discussed together, as they form both sides
of a single communication channel.

I have about a million questions, but I will start with three.

1. Once a client has established a connection with the server, I would
prefer to keep the connection open until they log out, which could be
several hours later. Is this an acceptable way of doing it, or should I keep
the number of open connections down to a minimum, closing and reopening as
required?

2. For authentication, the user enters a userid and password. The userid is
looked up in a database table. One of the columns contains the hashed (sha)
password. If the userid does not exist or the hashed password does not
match, I do not allow the user to proceed.

I looked at cred.py, but it seems more complicated to use this than stick to
my own method. Is this a valid observation?

A related question. With my present system, the authentication happens on
the client, so I do not have to send the password over the network. With the
new method, I will have to send it to the server. Should I be looking at
using SSL for protection against sniffing, or is there an easier way?

3. Whether or not I use cred, it looks as if I should be using Perspective
Broker (PB). I would normally have been thinking in terms of defining
message formats for exchanging data between server and client, so the
concept of defining objects that make calls on each other feels a bit
strange. If I maintain the full state of the connection on the server side,
I do not want to duplicate this on the client side, so I assume I must
define some form of stripped down object to represent the client state, and
use this to exchange information. Are there any simple examples I can look
at to explain this concept in more detail?

I feel I am at a crossroads. Either I can cobble something together that
works, but does not fit in with the Twisted philosophy, and will therefore
limit me as my application grows. Or I can get my mind more in tune with
Twisted, and set up structures that will work with me as things evolve.

If you have read this far, thanks for your perseverance. Any pointers or
suggestions will be much appreciated.

Thanks

Frank Millman