[Twisted-web] pgasync (was: /freeform_post!!random causes exceptions)

Andrea Arcangeli andrea at cpushare.com
Tue Jan 18 06:24:09 MST 2005


On Mon, Jan 17, 2005 at 07:53:02PM -0600, J Turner wrote:
> Yep, much was changed to make the API pain go away.

Indeed. I just happened to read it just a few days before ;).

> probably be wrapping a simple container of some kind around it in the
> future, that keeps the db args.

Ok fine. Otherwise I would have to do it in my app, and you can save
some code if you do it in the lib.

> > My suggestion is to make it backwards compatible with adbapi.  Isn't
> > that technically possible or am I missing something? I'd just like to
> > replace adbapi with pgasync and be done with it.
> > 
> > An API already exists, it's fine if you extend it to provide more
> > features and more finegrined access to the data, but I don't see why you
> > can't provide a backwards compatible API, which is also a nicer API IMHO.
> > 
> > If you can make it backwards compatible you've a chance to increase the
> > userbase quickly IMHO.
> 
> We had some talks about this in #twisted.web, and most seem to agree
> that this is true.  dialtone has been working on an adbapi wrapper, and
> it's very, very thin (like 30 lines) and seems to do the job.

This sounds great.

> I'm not a huge fan of *encouraging* the use of adbapi in new projects,
> however.  DB API is used by every single other project.  It's not really
> me that invented a new API.  :)

;) The point is that I hava to depend on the twisted API for my app, not
on DB API. There is no way I can handle an unstable API for the DB side
while developing my app, the overhead would be too huge, I don't have
time to maintain two APIs and adbapi is the only one I'm confortable to
depend right now. I choosed python only to go in production as fast as I
possibly can, I've truly no time to help on what's not strictly need for
my app, not more than answering these emails ;).

> Plus, I cannot make a true runInteraction, since the called function is
> assumed to be executed in a thread.  As far as runQuery() goes, that's
> basically connection.exFetch().  And runOperation is a trivial
> addition...

What's the difference between running it in a thread or not?

I've noticed myself a strange behaviour. If in the callback of adbapi I
run "raise 'something'", the exception handler page is not called before
rendering, but it's called in the middle of rendering. And in turn the
redirect to my /internal_server_error page doesn't trigger (the html is
interrupted in the middle, and the page is half rendered in the web
browser). So for now I'm catching the error when the db is disconnectd
and I print "no information available". For all other kind of exceptions
instead they jump immediatly to the internal_server_error. Is this
normal or is it a bug? Why isn't nevow waiting all deferred to complete,
before starting rendering the page? The deferred in question is the
value of an hash passed to the n:render="mapping" methid.

Something like this:

{ nr_accounts: d }

d is the deferred, and nevow starts rendering before waiting for it.
Then when d returns Failure("something"), it stops the page in the
middle. 

For now my workaround works. I know I'm going offtopic with pgasync. But
since you've mentioned the execution on a different thread I wondered
that it was related.

> > What about the type covnversions, does it gets everything right
> > automatically, like 'false' -> False, numeric -> decimal integer -> int
> > etc..?
> 
> It should!  protocol.py attempts to do all conversions based on the
> OIDs.  But if I missed something, it's a quick fix: let me know!

If you make the API compatible I'll give it a spin! But I need
runInteraction too ;) ;).

> > > When all is said and done, pgasync has, in every test I've tried, shown
> > > to be *much* faster than adbapi/threads/synclib in an async environment
> > > under heavy load. 
> > 
> > Most probably this is the removal of the futex locks (at least in
> > linux), I've run an strace on the main twisted thread, when it floods
> > data, and it's doing nothing but futex locking. Perhaps that could be
> > optimized a bit on the threaded version too. But I agree for a 2-way
> > pgasync is probably nicer in performance terms.
> 
> I don't know the exact reasons, but particularly under load (lots of
> concurrent connections/threads,) pgasync is much, much faster.  This is
> both on FreeBSD and Linux.

It's the thread locking. Try to strace the twisted main thread, and
you'll see the flood of FUTEX calls. As I've said I'd be surprised if
nothing can be done to improve it. But I certainly prefer pgasync
design myself too! But I need a compatible API since I need to go back
to adbapi if something goes wrong. I don't need performance right now.
I'll care about performance later. I'm looing forward to the day I will
see the website go down because of the too high load!! (well unless it
was a DDoS attack, ehehe ;)

Thanks for pgasync!



More information about the Twisted-web mailing list