[Twisted-Python] Scaling problem
tv at tv.debian.net
Wed Nov 5 11:14:08 EST 2003
Uwe C. Schroeder wrote:
> is there a way to have one service start several processes ?
> When using twisted everyone connecting to the server ends in the same process.
> This leads to "slowing down" when requesting information whenever a certain
> number of people go on. Each request takes some time to process and meanwhile
> the server is less responsive. I have one particularly expensive function
> that takes a bit. So if several people happen to use this function at a time
> the server slows down, up to where it takes a minute to complete.
> It's not a problem of system ressources (it's a quad processor xeon server
> with plenty of RAM and ultra 320 scsi drives running only this application
> and the postgres backend). The machine has a load of 0.2 while slowing down.
> Basically most of the machine is idle, since it only runs 2 processes and has
> 4 processors.
You should look at having a pool of worker _processes_, to which you
dispatch work with pb.
Not threads, as python threading will not really scale, and there's
really no point in threading if the amount of input/output to the worker
is small enough (shared mem has its good and bad sides).
> Is there a way to have twisted do something like apache or postgres and split
> the incoming connection to several processes ? It's not a web application,
> I'm solely using pb.
If you listen in a parent process, fork multiple children, and make the
children accept, unix does the right thing. I don't think Twisted knows
how to do that out of the box, but it should be doable.
That's assuming you don't need (much) shared data between the children.
> BTW: You might suggest to break down the expensive function into smaller
> pieces - no way. It has to run in one database transaction otherwise I might
> end up with a inconsistent database - no good in finance business.
If you are lucky, you can find a threading DB library (adbapi) that
actually scales. The other mail in this thread are more helpful there..
More information about the Twisted-Python