Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#4397 enhancement closed wontfix (wontfix)

wrap the asynchronous API of psycopg2 to provide a threadless replacement for ADBAPI

Reported by: Jan Urbański Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: Michael Tharp Branch:
Author:

Description

psycopg2, a Python driver for PostgreSQL recently added support for asynchronous connecting and querying using the underlying C library's asynchronous APIs. It would be cool to have a Twisted wrapper around it that would be more or less compatible with ADBAPI, but would not use threads.

The attached files provide such module along with tests. Compatibility with ADBAPI is quite good, but automatic reconnection is not implemented, because connecting is asynchronous and I felt it's better to leave the decision whether to reconnect to the user. Also, the connection pooling is so simplistic that I wouldn't be surprised if serious users would just write their own, I consider the existing one a PoC or example on how to go about doing it properly.

Attachments (5)

pgadbapi.py (17.9 KB) - added by Jan Urbański 9 years ago.
the PGADBAPI module
test_pgadbapi.py (21.1 KB) - added by Jan Urbański 9 years ago.
tests for PGADBAPI
test_pgadbapi.2.py (21.2 KB) - added by Jan Urbański 9 years ago.
unit tests doing skips if unable to connect to the db
pgadbapi.2.py (18.1 KB) - added by Jan Urbański 9 years ago.
adapt to the changed psycopg2 interface
test_pgadbapi.3.py (21.2 KB) - added by Jan Urbański 9 years ago.
psycopg2 changed the "issync()" method to the "async" property

Download all attachments as: .zip

Change History (16)

Changed 9 years ago by Jan Urbański

Attachment: pgadbapi.py added

the PGADBAPI module

Changed 9 years ago by Jan Urbański

Attachment: test_pgadbapi.py added

tests for PGADBAPI

comment:1 Changed 9 years ago by Jan Urbański

This code currently requires git HEAD of psycopg2 (not released yet) and some of my patches that were not merged yet. If anyone wants to play, the code for psycopg2 + my patches is at git://wulczer.org/psycopg2.git branch async_fixes (gitweb available)

comment:2 Changed 9 years ago by Jan Urbański

Keywords: review added
Owner: Glyph deleted

comment:3 Changed 9 years ago by Michael Tharp

Cc: Michael Tharp added

Changed 9 years ago by Jan Urbański

Attachment: test_pgadbapi.2.py added

unit tests doing skips if unable to connect to the db

comment:4 Changed 9 years ago by therve

It seems really interesting!

I'm not sure having it in Twisted is the best path, though. First most (all?) builders don't have postgres installed, so the test suite wouldn't run. Then I suspect the code is going to be a moving target for a bit, so you'll want to revert often. Also, relying on unreleased software in Twisted seems fishy.

What would you think of creating a project in the tx umbrella: https://launchpad.net/tx ?

comment:5 Changed 9 years ago by Jan Urbański

I hope some builders do, to test the ADBAPI code. The psycopg2 release containing the async stuff is going to happen in a few weeks at most, the API is being currently reviewed and will then be kept stable. It will probably change, but not in a dramatic way, from what I gathered from the mailing list.

If this won't find its way to Twisted I'm OK with putting it in tx.

comment:6 in reply to:  5 ; Changed 9 years ago by therve

Replying to wulczer:

I hope some builders do, to test the ADBAPI code.

*cough*

The psycopg2 release containing the async stuff is going to happen in a few weeks at most, the API is being currently reviewed and will then be kept stable. It will probably change, but not in a dramatic way, from what I gathered from the mailing list.

That's a good news.

If this won't find its way to Twisted I'm OK with putting it in tx.

It's just my opinion, I'd like to know what others think.

comment:7 in reply to:  6 Changed 9 years ago by Thijs Triemstra

Replying to therve:

It's just my opinion, I'd like to know what others think.

+1 for tx.

Changed 9 years ago by Jan Urbański

Attachment: pgadbapi.2.py added

adapt to the changed psycopg2 interface

comment:8 Changed 9 years ago by Jan Urbański

Attached is the hopefully final version that will be compatible with the soon-to-be-released psycopg2 version. The difference from the previous version is minimal, and the tests did not need to be changed. This depends on version 2f31a923e22e9982c16725a4fc2a158228658176 of my async_fixes branch.

comment:9 Changed 9 years ago by Jan Urbański

The asynchronous features just hit psycopg2 main branch (called python2). Attached is a one line modification of the tests due to the changed interface.

I also started a pgadbapi branch of Twisted on launchpad: lp:~wulczer/twisted/pgadbapi/

Changed 9 years ago by Jan Urbański

Attachment: test_pgadbapi.3.py added

psycopg2 changed the "issync()" method to the "async" property

comment:10 Changed 9 years ago by Itamar Turner-Trauring

Keywords: review removed
Resolution: wontfix
Status: newclosed

It's probably best if this was released as tx package, yes, both because it makes API changes and bug fix releases easier in the initial phases, and because it sounds like psycopg2 is still moving a target.

As a separate package, you'd presumably change your repository so this is no longer a sub-package of twisted, e.g. toplevel package pgadbapi (though that seems easy to typo) instead of twisted.pgadbapi.

So, I will close this ticket. Please post the final URL of your package to this ticket, to aid people who are googling and end up here. I look forward to seeing the release, it sounds pretty cool (blocking APIs are often annoying, especially when they don't do timeouts, *cough*Oracle*cough*).

comment:11 Changed 9 years ago by Jan Urbański

OK, I changed the name to txPostgres and put it Launchpad.

The final URL is https://launchpad.net/txpostgres.

Thanks for your time and interest. I hope TPPP (Twisted + Python + PostgreSQL + Psycopg2) will rock the world :o)

Note: See TracTickets for help on using tickets.