[Twisted-Python] adbapi / Postgres : LISTEN/NOTIFY and RealDictCursor

[Tobias Oberstein]

Hello Glyph,

thank's a lot for this advice! I thought I would have done right, and Twisted-style .. but the select FD trap wasn't on my radar.


The https://github.com/wulczer/txpostgres is very interesting. Guess I need to learn about interfaces.IReadWriteDescriptor
first to grasp it.

Regarding the broader picture: I cannot say I understand what you say completely.

" maximum file descriptor pollable by select()"

Is this *nix in general?

Does that only apply to select.select?

Or also: select.poll(), select.epoll(), select.kqueue() ?

How would implementing an IReadDescriptor circumvent the problem? By not using select(), but poll/epoll/kqeue?

In any case: we're running FreeBSD, so epoll() is out.

Is there even a kqueue-based Twisted reactor?

Sorry if helping me spawns new questions,

Thanks!
Tobias


for reference (don't know if it applies here):
psycopg since v2.2 has
http://initd.org/psycopg/docs/advanced.html#asynchronous-support


Von: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-bounces at twistedmatrix.com] Im Auftrag von Glyph
Gesendet: Donnerstag, 3. November 2011 23:54
An: Twisted general discussion
Betreff: Re: [Twisted-Python] adbapi / Postgres : LISTEN/NOTIFY and RealDictCursor

On Nov 3, 2011, at 6:18 PM, Tobias Oberstein wrote:


     while not self.stopped:
        if select.select([conn], [], [], self.timeout) == ([], [], []):
           pass


Please note that on high-volume servers, this is problematic, as 'conn' may be higher than the maximum file descriptor pollable by select().  So if you've tuned everything to be smart, use a high-volume reactor like epoll, and set all your ulimits correctly, you will still get tracebacks out of this code depending on random accidents of ordering.

This part of the logic might therefore be better implemented as a IReadDescriptor/IWriteDescriptor provider (i.e. a subclass of FileDescriptor), assuming you can ask your postgres binding to give you non-blocking reads and writes.  You're certainly not the first person to make this mistake though, <http://divillo.com/> gives the same advice :).

Also this would probably make a better patch for txpostgres <https://launchpad.net/txpostgres> than for Twisted itself, since support depends on specific database bindings.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20111103/7cb53792/attachment.html>