Ticket #1930 (new enhancement)

Opened 4 years ago

Last modified 2 years ago

create a libevent reactor

Reported by: PenguinOfDoom Owned by: therve
Priority: highest Milestone:
Component: core Keywords:
Cc: therve, antoine Branch: branches/libevent-1930-4
Author: therve Launchpad Bug:

Description

libevent provides primitives similar to addReader/addWriter/callLater. These could be used in a reactor.

Attachments

1930_libeventreactor_1.py Download (3.6 KB) - added by therve 4 years ago.
Basic implementation of libevent reactor
libevent.dll Download (48.0 KB) - added by bigdog 3 years ago.
windows libevent.dll built with visual studio 2003
_libevent.pyd Download (27.5 KB) - added by therve 3 years ago.
test.patch Download (9.7 KB) - added by antoine 3 years ago.
libevent-gc.patch Download (17.0 KB) - added by antoine 3 years ago.

Change History

  Changed 4 years ago by exarkun

  • owner changed from glyph to PenguinOfDoom
  • priority changed from normal to low

Changed 4 years ago by therve

Basic implementation of libevent reactor

  Changed 4 years ago by therve

I've tried to make a libevent reactor, but I'm a bit stuck for now (some segfaults appear in the tests). I attach my current work if somebody is interested.

  Changed 4 years ago by therve

  • cc therve added

  Changed 4 years ago by therve

OK I've found the problem in libevent/eventmodule.c, in function __libevent_ev_callback (a problem with reference counting).

I have a fix, but I'm waiting for the livent-python to be up to see if it's not already corrected in the trunk. I'll send it to anyone interested.

  Changed 4 years ago by therve

It seems lots of work has been done here: source:/branches/slyphon/libevent-4/, using the libevent C bindings directly with Pyrex (I used the python bindings), and implementing callLater (I just did the FD management).

Changed 3 years ago by bigdog

windows libevent.dll built with visual studio 2003

  Changed 3 years ago by bigdog

If you need to build libevent on windows

1) #include <winsock2.h> must appear before #include<windows.h>

2) windows does not support function , create a macro that defined function

3) add #pragma to pull in socket lib (not sure if this is required for use in twisted/python. It was required to link the test app.

Changed 3 years ago by therve

  Changed 3 years ago by therve

I attached a build of the module for windows. I had to do some little hack to make it build. The tests doesn't pass for now, I'll probably need some help for this.

Oh, and I think using libevent in Windows use select, some that may not be that interesting (for now at least).

  Changed 3 years ago by therve

  • keywords review added
  • owner PenguinOfDoom deleted
  • priority changed from low to highest

The branch libevent-1930-2 is in pretty good shape. The main missing thing is Windows support, but it would need some Windows hero.

  Changed 3 years ago by exarkun

  • owner set to therve
  • keywords review removed
  • test_libevent
    • test_tim eout is bad :/ I'm not really sure what should be done about it, but something needs to be. real-time using tests will fail spuriously. test_loop has a similar timing issue, and is also quite long.
    • test_signalHandler is scary.
  • libeventreactor.py
    • hooray per-instance state
    • reads/writes/selectables should probably be private. they should be doc'd, too. if you want to copy some strings out of the closed-socket-1537-3 branch for this, that's cool. I imagine some refactoring is on the horizon.
  • _libevent.c
    • I think EventBase_New leaks memory when event_init fails.
    • I think EventBase_CreateEvent leaks memory when either Event_Init or PyObject_CallMethod("setEventBase") fails.
    • __libevent_ev_callback calls several Python APIs (Py_DECREF, Py_CLEAR, PyErr_WriteUnraisable) without holding the interpreter lock. The comment about how exceptions are handled has the right general idea, but neither WriteUnraisable nor SetAsyncExc is proper. I think terminating the event loop actually is the right thing to do. If I understand correctly, this will actually only cause dispatch() to raise. A reactor implemented with this module can handle the exception there and then call dispatch again. A first pass of this reactor might continue to use PyErr_WriteUnraisable (albeit correctly), but if there's no pressing need to merge this branch (as I think there isn't), doing it the right way now might make sense.
    • In Event_SetEventBase, Py_XDECREF is used after a NULL check. Might as well drop the check and use Py_DECREF, since that's the only difference between the two.
    • There might be something fishy about the order of execution at the end of Event_SetEventBase. I wonder if decref'ing the existing eventBase could ever result in the new eventBase being freed. If the existing eventBase held the only reference to the new one, and nothing else held a reference to the existing eventBase, then the new eventBase would be freed immediately before the Py_INCREF call saves it. Then the function would probably crash.

  Changed 3 years ago by therve

(In [21069])

  • Remote time related tests
  • Add some missing decrefs
  • Cleanups

Refs #1930

  Changed 3 years ago by therve

(In [22068]) Remove a stupid test, some cleanups.

Refs #1930

  Changed 3 years ago by therve

  • branch set to branches/libevent-1930-3
  • author set to therve

(In [22069]) Branching to 'libevent-1930-3'

  Changed 3 years ago by antoine

Hi

since I wanted something fun to do I created some tests to check for refcount problems. Attaching patch: there are some failures :-)

  Changed 3 years ago by therve

(In [22263]) Apply patch from antoine, adding failing tests for refcounting.

Refs #1930

  Changed 3 years ago by antoine

Oops, I have attached an updated version just before you committed the patch. There are a few more tests for persistent events.

  Changed 3 years ago by antoine

After playing a bit, it seems the flag EV_PERSIST does not make sense for pure timers, so the persist parameter to createTimer should probably be removed. Unless I've missed something :)

Changed 3 years ago by antoine

  Changed 3 years ago by antoine

I take back what I said, EV_PERSIST does make sense for "timers" because a timer is actually the loop timeout event. Sorry for posting too quick.

Changed 3 years ago by antoine

  Changed 3 years ago by antoine

Here is a new patch against the current branch contents. It contains additional tests and also modifies _libevent.c so that all tests pass (by enabling proper garbage collection of Event and EventBase objects, and maintaining internal structures to know which events are registered).

Unfortunately there is a remaining problem I haven't managed to fix: if I enable event_base_free(...) in the EventBase deallocator, some tests fail with an assertion error in libevent. This deallocation call wasn't present in the original version either.

  Changed 3 years ago by antoine

  • cc antoine added

  Changed 3 years ago by therve

(In [22264]) Apply another patch, solving previous problems, with more tests.

Refs #1930

  Changed 3 years ago by therve

(In [22265]) Not all platforms raise EINPROGRESS, unfortunately.

Refs #1930

follow-up: ↓ 24   Changed 3 years ago by therve

Thanks a lot for the patch. I wasa wondering if we could use the libevent internal structures, and not storing the events on our own. But for now it does the trick.

  Changed 3 years ago by therve

(In [22266]) Remove some time related failures, cleanups.

Refs #1930

in reply to: ↑ 22   Changed 3 years ago by antoine

Replying to therve:

Thanks a lot for the patch. I wasa wondering if we could use the libevent internal structures, and not storing the events on our own. But for now it does the trick.

We could get rid of expiredEvents:

  • either by trusting libevent not to use the event struct once the callback returns (this is true in the current libevent version, 1.4.1-beta), so we can directly decref the EventObject in the callback
  • or by using event_base_once() for non-persistent objects, but this is incompatible with the current Python API (which separates createEvent() and addToLoop()).

  Changed 3 years ago by therve

(In [22320]) Use Py_VISIT in traverse methods

Refs #1930

  Changed 2 years ago by therve

  • branch changed from branches/libevent-1930-3 to branches/libevent-1930-4

(In [23487]) Branching to 'libevent-1930-4'

Note: See TracTickets for help on using tickets.