create a libevent reactor

[PenguinOfDoom]

libevent provides primitives similar to addReader/addWriter/callLater. These could be used in a reactor.

[therve]

Basic implementation of libevent reactor

[therve]

I've tried to make a libevent reactor, but I'm a bit stuck for now (some segfaults appear in the tests). I attach my current work if somebody is interested.

[therve]

OK I've found the problem in libevent/eventmodule.c, in function __libevent_ev_callback (a problem with reference counting).

I have a fix, but I'm waiting for the livent-python to be up to see if it's not already corrected in the trunk. I'll send it to anyone interested.

[therve]

It seems lots of work has been done here: source:/branches/slyphon/libevent-4/, using the libevent C bindings directly with Pyrex (I used the python bindings), and implementing callLater (I just did the FD management).

[bigdog]

windows libevent.dll built with visual studio 2003

[bigdog]

If you need to build libevent on windows

1) #include <winsock2.h> must appear before #include<windows.h>

2) windows does not support function , create a macro that defined function

3) add #pragma to pull in socket lib (not sure if this is required for use in twisted/python. It was required to link the test app.

[therve]

I attached a build of the module for windows. I had to do some little hack to make it build. The tests doesn't pass for now, I'll probably need some help for this.

Oh, and I think using libevent in Windows use select, some that may not be that interesting (for now at least).

[therve]

The branch libevent-1930-2 is in pretty good shape. The main missing thing is Windows support, but it would need some Windows hero.

[exarkun]

• test_libevent
  • test_tim eout is bad :/ I'm not really sure what should be done about it, but something needs to be. real-time using tests will fail spuriously. test_loop has a similar timing issue, and is also quite long.
  • test_signalHandler is scary.
• libeventreactor.py
  • hooray per-instance state
  • reads/writes/selectables should probably be private. they should be doc'd, too. if you want to copy some strings out of the closed-socket-1537-3 branch for this, that's cool. I imagine some refactoring is on the horizon.
• _libevent.c
  • I think EventBase_New leaks memory when event_init fails.
  • I think EventBase_CreateEvent leaks memory when either Event_Init or PyObject_CallMethod("setEventBase") fails.
  • __libevent_ev_callback calls several Python APIs (Py_DECREF, Py_CLEAR, PyErr_WriteUnraisable) without holding the interpreter lock. The comment about how exceptions are handled has the right general idea, but neither WriteUnraisable? nor SetAsyncExc? is proper. I think terminating the event loop actually is the right thing to do. If I understand correctly, this will actually only cause dispatch() to raise. A reactor implemented with this module can handle the exception there and then call dispatch again. A first pass of this reactor might continue to use PyErr_WriteUnraisable (albeit correctly), but if there's no pressing need to merge this branch (as I think there isn't), doing it the right way now might make sense.
  • In Event_SetEventBase, Py_XDECREF is used after a NULL check. Might as well drop the check and use Py_DECREF, since that's the only difference between the two.
  • There might be something fishy about the order of execution at the end of Event_SetEventBase. I wonder if decref'ing the existing eventBase could ever result in the new eventBase being freed. If the existing eventBase held the only reference to the new one, and nothing else held a reference to the existing eventBase, then the new eventBase would be freed immediately before the Py_INCREF call saves it. Then the function would probably crash.

[therve]

(In [21069])

• Remote time related tests
• Add some missing decrefs
• Cleanups

Refs #1930

[therve]

(In [22068]) Remove a stupid test, some cleanups.

Refs #1930

[therve]

(In [22069]) Branching to 'libevent-1930-3'

[antoine]

Hi

since I wanted something fun to do I created some tests to check for refcount problems. Attaching patch: there are some failures :-)

[therve]

(In [22263]) Apply patch from antoine, adding failing tests for refcounting.

Refs #1930

[antoine]

Oops, I have attached an updated version just before you committed the patch. There are a few more tests for persistent events.

[antoine]

After playing a bit, it seems the flag EV_PERSIST does not make sense for pure timers, so the persist parameter to createTimer should probably be removed. Unless I've missed something :)

[antoine]

I take back what I said, EV_PERSIST does make sense for "timers" because a timer is actually the loop timeout event. Sorry for posting too quick.

[antoine]

Here is a new patch against the current branch contents. It contains additional tests and also modifies _libevent.c so that all tests pass (by enabling proper garbage collection of Event and EventBase? objects, and maintaining internal structures to know which events are registered).

Unfortunately there is a remaining problem I haven't managed to fix: if I enable event_base_free(...) in the EventBase? deallocator, some tests fail with an assertion error in libevent. This deallocation call wasn't present in the original version either.

[therve]

(In [22264]) Apply another patch, solving previous problems, with more tests.

Refs #1930

[therve]

(In [22265]) Not all platforms raise EINPROGRESS, unfortunately.

Refs #1930

[therve]

Thanks a lot for the patch. I wasa wondering if we could use the libevent internal structures, and not storing the events on our own. But for now it does the trick.

[therve]

(In [22266]) Remove some time related failures, cleanups.

Refs #1930

[antoine]

Replying to therve:

Thanks a lot for the patch. I wasa wondering if we could use the libevent internal structures, and not storing the events on our own. But for now it does the trick.

We could get rid of expiredEvents:

• either by trusting libevent not to use the event struct once the callback returns (this is true in the current libevent version, 1.4.1-beta), so we can directly decref the EventObject? in the callback
• or by using event_base_once() for non-persistent objects, but this is incompatible with the current Python API (which separates createEvent() and addToLoop()).

[therve]

(In [22320]) Use Py_VISIT in traverse methods

Refs #1930

[therve]

(In [23487]) Branching to 'libevent-1930-4'