[Twisted-web] pgasync (was: /freeform_post!!random causes exceptions)

[J Turner]

On Fri, Jan 14, 2005 at 07:16:27PM +0100, Andrea Arcangeli wrote:
> On Thu, Jan 13, 2005 at 01:32:51PM -0600, J Turner wrote:
> > On Mon, Jan 10, 2005 at 06:03:40PM +0100, Andrea Arcangeli wrote:
> You must have changed quite some bits in the meantime, thanks.

Yep, much was changed to make the API pain go away.

> Still I'm not convinced creating a new connection (even if you
> internally do pooling) is the right API.  I'm also not sure if you've a
> maximum number of open connections. The thread limited the number of
> connections. I don't dislike having a max number too (but of course it's
> a very minor feature).
> 
> The part I like less in the API is the pgasync.connect(**DBARGS) with
> DBARGS being a global (especially that a gloabl with username and pwd
> inside doesn't look very nice IMHO ;). But I can fix this on my side by
> hiding it in my own sql_class, but personally I prefer the idea of
> passing the username and password to a "pooling class" during startup,
> and then to do the operations at runtime on the pooling class.

Yeah, I agree.  I'm not completely sold on that part either.  It's the
awkward bit that snuck out of the "meet db api halfway" thing.  I will
probably be wrapping a simple container of some kind around it in the
future, that keeps the db args.

> > But feel free to offer suggestions, I'm all about improving this.
> 
> My suggestion is to make it backwards compatible with adbapi.  Isn't
> that technically possible or am I missing something? I'd just like to
> replace adbapi with pgasync and be done with it.
> 
> An API already exists, it's fine if you extend it to provide more
> features and more finegrined access to the data, but I don't see why you
> can't provide a backwards compatible API, which is also a nicer API IMHO.
> 
> If you can make it backwards compatible you've a chance to increase the
> userbase quickly IMHO.

We had some talks about this in #twisted.web, and most seem to agree
that this is true.  dialtone has been working on an adbapi wrapper, and
it's very, very thin (like 30 lines) and seems to do the job.

I'm not a huge fan of *encouraging* the use of adbapi in new projects,
however.  DB API is used by every single other project.  It's not really
me that invented a new API.  :)

Plus, I cannot make a true runInteraction, since the called function is
assumed to be executed in a thread.  As far as runQuery() goes, that's
basically connection.exFetch().  And runOperation is a trivial
addition...

> What about the type covnversions, does it gets everything right
> automatically, like 'false' -> False, numeric -> decimal integer -> int
> etc..?

It should!  protocol.py attempts to do all conversions based on the
OIDs.  But if I missed something, it's a quick fix: let me know!

> > When all is said and done, pgasync has, in every test I've tried, shown
> > to be *much* faster than adbapi/threads/synclib in an async environment
> > under heavy load. 
> 
> Most probably this is the removal of the futex locks (at least in
> linux), I've run an strace on the main twisted thread, when it floods
> data, and it's doing nothing but futex locking. Perhaps that could be
> optimized a bit on the threaded version too. But I agree for a 2-way
> pgasync is probably nicer in performance terms.

I don't know the exact reasons, but particularly under load (lots of
concurrent connections/threads,) pgasync is much, much faster.  This is
both on FreeBSD and Linux.

 - jamwt