[Twisted-Python] adbapi / Postgres : LISTEN/NOTIFY and RealDictCursor

Tobias Oberstein tobias.oberstein at tavendo.de
Thu Nov 3 19:40:04 EDT 2011

Hello Glyph,

thank's a lot for this advice! I thought I would have done right, and Twisted-style .. but the select FD trap wasn't on my radar.

The https://github.com/wulczer/txpostgres is very interesting. Guess I need to learn about interfaces.IReadWriteDescriptor
first to grasp it.

Regarding the broader picture: I cannot say I understand what you say completely.

" maximum file descriptor pollable by select()"

Is this *nix in general?

Does that only apply to select.select?

Or also: select.poll(), select.epoll(), select.kqueue() ?

How would implementing an IReadDescriptor circumvent the problem? By not using select(), but poll/epoll/kqeue?

In any case: we're running FreeBSD, so epoll() is out.

Is there even a kqueue-based Twisted reactor?

Sorry if helping me spawns new questions,


for reference (don't know if it applies here):
psycopg since v2.2 has

Von: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-bounces at twistedmatrix.com] Im Auftrag von Glyph
Gesendet: Donnerstag, 3. November 2011 23:54
An: Twisted general discussion
Betreff: Re: [Twisted-Python] adbapi / Postgres : LISTEN/NOTIFY and RealDictCursor

On Nov 3, 2011, at 6:18 PM, Tobias Oberstein wrote:

     while not self.stopped:
        if select.select([conn], [], [], self.timeout) == ([], [], []):

Please note that on high-volume servers, this is problematic, as 'conn' may be higher than the maximum file descriptor pollable by select().  So if you've tuned everything to be smart, use a high-volume reactor like epoll, and set all your ulimits correctly, you will still get tracebacks out of this code depending on random accidents of ordering.

This part of the logic might therefore be better implemented as a IReadDescriptor/IWriteDescriptor provider (i.e. a subclass of FileDescriptor), assuming you can ask your postgres binding to give you non-blocking reads and writes.  You're certainly not the first person to make this mistake though, <http://divillo.com/> gives the same advice :).

Also this would probably make a better patch for txpostgres <https://launchpad.net/txpostgres> than for Twisted itself, since support depends on specific database bindings.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20111103/7cb53792/attachment.htm 

More information about the Twisted-Python mailing list