Opened 12 years ago

Closed 7 years ago

#1083 defect closed wontfix (wontfix)

gtk2reactor thread initialization confuses python-2.3

Reported by: loic Owned by:
Priority: low Milestone:
Component: core Keywords: documentation
Cc: Jean-Paul Calderone, loic Branch:


Attachments (1)

trace (6.6 KB) - added by loic 12 years ago.

Download all attachments as: .zip

Change History (21)

comment:1 Changed 12 years ago by loic

I do the following. Run a gtk2reactor based python application
that calls a C function that calls a python function. When the
python function is called, python crashes because its thread state
variable is invalid. 

I commented the thread initialization lines in and
everything works fine. 

I realize this bug report is too terse to be truly useful. However,
I first wanted to make sure this problem was not already known in
another form (I searched the database and did not find anything 
obviously related). 

My hunch is that something bad happens when glib initializes threads
and python also tries to do thread related initialization. 

If the problem has no obvious solution I will submit a more detailed
bug report with version numbers for glib/python/twisted etc. on my
Debian GNU/Linux box running sid.

==9216== Process terminating with default action of signal 11 (SIGSEGV)
==9216==  Access not within mapped region at address 0x8
==9216==    at 0x80EF002: PyFrame_New (frameobject.c:540)
==9216==    by 0x80AD035: PyEval_EvalCodeEx (ceval.c:2461)
==9216==    by 0x80F0B77: function_call (funcobject.c:504)

comment:2 Changed 12 years ago by loic

If I am not mistaken, gtk (gdk really) uses xthreads on my 
GNU/Linux Debian sid box but python2.3 is based on pthreads.

loic@allin:/tmp$ pkg-config --cflags gdk-x11-2.0
-DXTHREADS -I/usr/include/gtk-2.0 -I/usr/lib/gtk-2.0/include
-I/usr/X11R6/include -I/usr/include/pango-1.0 -I/usr/include/freetype2
-I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include  

loic@allin:/tmp$ ldd /usr/bin/python2.3 => /lib/tls/ (0xb7fc4000)

I will follow this lead if noone tells me that it is a dead end.

comment:3 Changed 12 years ago by loic

Here is a simple workaround that applications having the same 
problem can use:

import gobject
if hasattr(gobject, "threads_init"):
    del gobject.threads_init

This is ugly.

comment:4 Changed 12 years ago by Jean-Paul Calderone

This is most likely a bug in PyGTK.  Could you be a bit more specific in how you
produce this failure, though?  It's not very clear what sequence events is
occuring: when does the reactor start, when does your code run, when does the
extension code call back into Python, when are threads started, when is PyGTK's
threading system initialized?

The story with threads in PyGTK is basically that you *must* initialize PyGTK's
threading system before starting any threads.  If you do not do this, the
interpreter will crash, as you have observed.  There's nothing Twisted can do
about this (and seemingly nothing the PyGTK developers want to do about it).

Perhaps we could document this PyGTK shortcoming in the gtk2reactor docs, though.

comment:5 Changed 12 years ago by loic

The usage sequence that is triggering the problem is as follows:

- python where contains 
from twisted.internet import gtk2reactor
Actual source at

- At some point the python script will call a python function implemented
  in a python C module. 
  The python C module is at

- The function of the python C module will call a callback function with

  PyObject* tuple = PyList_AsTuple(out_stream);
  g_assert(PyList_SetSlice(out_stream, 0, PyList_Size(out_stream), NULL) == 0);
    PyObject* result = PyObject_Call(callback, tuple, NULL);
    if(result) { Py_DECREF(result); }

The crash always occurs when PyObject_Call is called.

In other words the sequence is Python -> Python C module -> Python. 

A few words about the context:

- valgrind is perfectly happy with everything (no warning whatsoever, safe
  for one related to X11 library reading 4 bytes in uninitialized memory) 
  until the program crashes with the memory access violation described
  in the report.

- The program makes absolutely no explicit use of threads anywhere.

- The versions in use are:
libgtk 2.0-0 
twisted 2.0.1-3
libglade 2.5.1-2
python-gtk2-dev 2.6.2-1
python2.3 2.3.5-4

Changed 12 years ago by loic

Attachment: trace added

comment:6 Changed 12 years ago by loic

More information about this bug. When I apply the workaround suggested below,
DNS name resolution hangs. The stack traces are as included in the attached file.

comment:7 Changed 12 years ago by loic

Here is a workaround to avoid the bug (including the workaround mentioned in
an earlier message) that can fit in code made to also run with twisted-1.3.
The default resolver creates a thread that blocks completly.

import gobject
if hasattr(gobject, "threads_init"):
    del gobject.threads_init

from twisted.internet import gtk2reactor
    from twisted.internet.base import BlockingResolver

comment:8 Changed 12 years ago by Jean-Paul Calderone

Is flush_io_channel() called while the interpreter lock is held?  It does not
appear so to me.  If not, it will need to acquire the lock before calling any
Python APIs.  The way to do this is with PyGILState_Ensure.  Afterwards the lock
should be released with PyGILState_Release.

If you make these changes, does the app still crash with an unmodified gtk2reactor?

comment:9 Changed 12 years ago by loic

I was not aware of the PyGILState* functions. This, indeed, solves
the initial problem reported. Excellent. However, the non blocking DNS issue
remains. I'll make a simple test case that demonstrates it. We are making
progess :-)

comment:10 Changed 12 years ago by Jean-Paul Calderone

Any progress on this?

comment:11 Changed 12 years ago by loic

Not yet. I've been trying to reproduce the problem with a minimalist
code but it does not show. I am in the process of cutting everything I
can in my code and see when the problem disapear. Hopefully it is a mistake
I did. But I am not sure yet since I do not manipulate threads at all.

comment:12 Changed 11 years ago by Jean-Paul Calderone

Any new information here?

comment:13 Changed 11 years ago by loic

Jp Calderone \[Twisted issue tracker\] writes:
 > Jp Calderone <> added the comment:
 > Any new information here?

        Not yet. I've lived so far with the blocking dns resolution to
avoid the problem. But blocking dns resolution started to be a problem
therefore I think I'll have to get back to fixing this problem :-(


comment:14 Changed 10 years ago by letrigre

It works with python2.4

comment:15 in reply to:  14 Changed 10 years ago by letrigre

Resolution: fixed
Status: newclosed

Replying to letrigre:

It works with python2.4 with twisted 2.4

comment:16 Changed 10 years ago by Jean-Paul Calderone

Resolution: fixed
Status: closedreopened

Which may be why the ticket is about Python 2.3...

comment:17 Changed 9 years ago by Jean-Paul Calderone

Priority: highlow

Well, I still don't have any particular insight here. It's not likely I'll be fixing this any time soon. If someone else wants to step up, please do, otherwise it looks like this will just linger until we drop Python 2.3 support, and then be closed as wontfix. ;)

comment:18 Changed 9 years ago by hypatia

Cc: hypatia removed

comment:19 Changed 7 years ago by Rotund

Resolution: wontfix
Status: reopenedclosed

According to the comments, this worked on 2.4 but not on Python 2.3, which is no longer supported. Closing bug as will not fix.

comment:20 Changed 6 years ago by <automation>

Owner: Jean-Paul Calderone deleted
Note: See TracTickets for help on using tickets.