Opened 10 years ago

Closed 5 years ago

#1083 defect closed wontfix (wontfix)

gtk2reactor thread initialization confuses python-2.3

Reported by: loic Owned by:
Priority: low Milestone:
Component: core Keywords: documentation
Cc: exarkun, loic Branch:
Author: Launchpad Bug:

Description


Attachments (1)

trace (6.6 KB) - added by loic 9 years ago.

Download all attachments as: .zip

Change History (21)

comment:1 Changed 10 years ago by loic

I do the following. Run a gtk2reactor based python application
that calls a C function that calls a python function. When the
python function is called, python crashes because its thread state
variable is invalid. 

I commented the thread initialization lines in gtk2reactor.py and
everything works fine. 

I realize this bug report is too terse to be truly useful. However,
I first wanted to make sure this problem was not already known in
another form (I searched the database and did not find anything 
obviously related). 

My hunch is that something bad happens when glib initializes threads
and python also tries to do thread related initialization. 

If the problem has no obvious solution I will submit a more detailed
bug report with version numbers for glib/python/twisted etc. on my
Debian GNU/Linux box running sid.


==9216== Process terminating with default action of signal 11 (SIGSEGV)
==9216==  Access not within mapped region at address 0x8
==9216==    at 0x80EF002: PyFrame_New (frameobject.c:540)
==9216==    by 0x80AD035: PyEval_EvalCodeEx (ceval.c:2461)
==9216==    by 0x80F0B77: function_call (funcobject.c:504)

comment:2 Changed 10 years ago by loic

If I am not mistaken, gtk (gdk really) uses xthreads on my 
GNU/Linux Debian sid box but python2.3 is based on pthreads.

loic@allin:/tmp$ pkg-config --cflags gdk-x11-2.0
-DXTHREADS -I/usr/include/gtk-2.0 -I/usr/lib/gtk-2.0/include
-I/usr/X11R6/include -I/usr/include/pango-1.0 -I/usr/include/freetype2
-I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include  

loic@allin:/tmp$ ldd /usr/bin/python2.3
		libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7fc4000)

I will follow this lead if noone tells me that it is a dead end.

comment:3 Changed 10 years ago by loic

Here is a simple workaround that applications having the same 
problem can use:

import gobject
if hasattr(gobject, "threads_init"):
    del gobject.threads_init

This is ugly.

comment:4 Changed 10 years ago by exarkun

This is most likely a bug in PyGTK.  Could you be a bit more specific in how you
produce this failure, though?  It's not very clear what sequence events is
occuring: when does the reactor start, when does your code run, when does the
extension code call back into Python, when are threads started, when is PyGTK's
threading system initialized?

The story with threads in PyGTK is basically that you *must* initialize PyGTK's
threading system before starting any threads.  If you do not do this, the
interpreter will crash, as you have observed.  There's nothing Twisted can do
about this (and seemingly nothing the PyGTK developers want to do about it).

Perhaps we could document this PyGTK shortcoming in the gtk2reactor docs, though.

comment:5 Changed 10 years ago by loic

The usage sequence that is triggering the problem is as follows:

- python script.py where script.py contains 
from twisted.internet import gtk2reactor
gtk2reactor.install()
Actual source at
http://cvs.sourceforge.net/viewcvs.py/pokersource/poker-network/pokerclient2d/poker2d.py?rev=1.8&view=auto

- At some point the python script will call a python function implemented
  in a python C module. 
  The python C module is at
http://cvs.sourceforge.net/viewcvs.py/pokersource/poker-network/pokerclient2d/python.c?rev=1.4&view=auto

- The function of the python C module will call a callback function with

  PyObject* tuple = PyList_AsTuple(out_stream);
  g_assert(tuple);
  g_assert(PyList_SetSlice(out_stream, 0, PyList_Size(out_stream), NULL) == 0);
  {
    PyObject* result = PyObject_Call(callback, tuple, NULL);
    if(result) { Py_DECREF(result); }
  }
  Py_DECREF(tuple);

The crash always occurs when PyObject_Call is called.

In other words the sequence is Python -> Python C module -> Python. 

A few words about the context:

- valgrind is perfectly happy with everything (no warning whatsoever, safe
  for one related to X11 library reading 4 bytes in uninitialized memory) 
  until the program crashes with the memory access violation described
  in the report.

- The program makes absolutely no explicit use of threads anywhere.

- The versions in use are:
libgtk 2.0-0 
twisted 2.0.1-3
libglade 2.5.1-2
python-gtk2-dev 2.6.2-1
python2.3 2.3.5-4

Changed 9 years ago by loic

comment:6 Changed 9 years ago by loic

More information about this bug. When I apply the workaround suggested below,
DNS name resolution hangs. The stack traces are as included in the attached file.

comment:7 Changed 9 years ago by loic

Here is a workaround to avoid the bug (including the workaround mentioned in
an earlier message) that can fit in code made to also run with twisted-1.3.
The default resolver creates a thread that blocks completly.

import gobject
if hasattr(gobject, "threads_init"):
    del gobject.threads_init

from twisted.internet import gtk2reactor
gtk2reactor.install()
try:
    from twisted.internet.base import BlockingResolver
    reactor.installResolver(BlockingResolver())
except:
    pass

comment:8 Changed 9 years ago by exarkun

Is flush_io_channel() called while the interpreter lock is held?  It does not
appear so to me.  If not, it will need to acquire the lock before calling any
Python APIs.  The way to do this is with PyGILState_Ensure.  Afterwards the lock
should be released with PyGILState_Release.

If you make these changes, does the app still crash with an unmodified gtk2reactor?

comment:9 Changed 9 years ago by loic

I was not aware of the PyGILState* functions. This, indeed, solves
the initial problem reported. Excellent. However, the non blocking DNS issue
remains. I'll make a simple test case that demonstrates it. We are making
progess :-)

comment:10 Changed 9 years ago by exarkun

Any progress on this?

comment:11 Changed 9 years ago by loic

Not yet. I've been trying to reproduce the problem with a minimalist
code but it does not show. I am in the process of cutting everything I
can in my code and see when the problem disapear. Hopefully it is a mistake
I did. But I am not sure yet since I do not manipulate threads at all.

comment:12 Changed 9 years ago by exarkun

Any new information here?

comment:13 Changed 9 years ago by loic

Jp Calderone \[Twisted issue tracker\] writes:
 > 
 > Jp Calderone <exarkun@twistedmatrix.com> added the comment:
 > 
 > Any new information here?

        Not yet. I've lived so far with the blocking dns resolution to
avoid the problem. But blocking dns resolution started to be a problem
therefore I think I'll have to get back to fixing this problem :-(

        Cheers,

comment:14 follow-up: Changed 8 years ago by letrigre

It works with python2.4

comment:15 in reply to: ↑ 14 Changed 8 years ago by letrigre

  • Resolution set to fixed
  • Status changed from new to closed

Replying to letrigre:

It works with python2.4 with twisted 2.4

comment:16 Changed 8 years ago by exarkun

  • Resolution fixed deleted
  • Status changed from closed to reopened

Which may be why the ticket is about Python 2.3...

comment:17 Changed 7 years ago by exarkun

  • Priority changed from high to low

Well, I still don't have any particular insight here. It's not likely I'll be fixing this any time soon. If someone else wants to step up, please do, otherwise it looks like this will just linger until we drop Python 2.3 support, and then be closed as wontfix. ;)

comment:18 Changed 7 years ago by hypatia

  • Cc hypatia removed

comment:19 Changed 5 years ago by Rotund

  • Resolution set to wontfix
  • Status changed from reopened to closed

According to the comments, this worked on 2.4 but not on Python 2.3, which is no longer supported. Closing bug as will not fix.

comment:20 Changed 4 years ago by <automation>

  • Owner exarkun deleted
Note: See TracTickets for help on using tickets.