[Twisted-Python] intermittent traceback using adbapi with pyPgSQL

Stephen Waterbury golux at comcast.net
Fri Aug 6 14:58:18 EDT 2004

Twisted gurus,

I didn't want to create a bug report yet
because I'm not sure that it's a Twisted bug.  I
have only observed it while using pyPgSQL as the adaptor;
haven't seen it yet (and have run the exact same code
that produces it) with psycopg.

I see this traceback intermittently occur at the end of
chains of deferreds that include 2 or more adbapi
calls.  I can provide some example code, but I should
really develop a concise example (as opposed to my app code,
which is spread over 2 or 3 modules) that might be useful
as the basis for a unit test in case this is a Twisted
problem -- it's not clear to me where the problem is:
my code, Twisted, or pyPgSQL.  My natural assumption
would be the former, but the intermittency and the
adaptor-dependence makes me wonder.

I can run the exact same code several times without
incident and then suddenly get this traceback, which makes
me think it might be some allocated limit being exceeded
(threads in the thread pool?  sorry, clueless here).

Some observations:

* all database operations in the deferred chain complete
   successfully before the traceback occurs.

* it only occurs with pyPgSQL -- not with psycopg
   (at least I haven't yet seen it with psycopg, and
   believe me I've tried to make it happen! ... perhaps
   psycopg's thread management prevents it?)

* this always occurs at the end of a "test" in which there
   is a succeed(result) callback that prints the result and
   calls reactor.stop() -- I wonder if stopping the reactor
   (which wouldn't normally happen in an app, of course)
   breaks the pipe?

* I'm using
   - PostgreSQL 7.4.2
   - pyPgSQL 2.4
   - psycopg 1.1.15
   - Twisted 1.3.0
   - Python Python 2.3.3 (#1, Jul 22 2004, 15:58:51)
     [GCC 3.3.2 20031022 (Red Hat Linux 3.3.2-1)] on linux2
   - Fedora Core 1,  2.4.22-1.2197.nptl

Anyway, here's the traceback:
Traceback (most recent call last):
   File "/usr/local/lib/python2.3/threading.py", line 436, in __bootstrap
   File "/usr/local/lib/python2.3/threading.py", line 416, in run
     self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
   File "/usr/local/lib/python2.3/site-packages/twisted/python/threadpool.py", 
line 157, in _worker
     context.call(ctx, function, *args, **kwargs)
   File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 
64, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 
43, in callWithContext
     return func(*args,**kw)
   File "/usr/local/lib/python2.3/site-packages/twisted/internet/threads.py", 
line 42, in _putResultInDeferred
     reactor.callFromThread(deferred.callback, result)
   File "/usr/local/lib/python2.3/site-packages/twisted/internet/base.py", line 
201, in callFromThread
   File "/usr/local/lib/python2.3/site-packages/twisted/internet/base.py", line 
207, in wakeUp
   File "/usr/local/lib/python2.3/site-packages/twisted/internet/default.py", 
line 399, in wakeUp
exceptions.IOError: [Errno 32] Broken pipe

Has anyone else observed this?


More information about the Twisted-Python mailing list