[Twisted-Python] win32 reactors

glyph at divmod.com glyph at divmod.com
Mon Mar 31 21:21:37 EDT 2008

On 31 Mar, 10:15 pm, james at mansionfamily.plus.com wrote:
>glyph at divmod.com wrote:

>>These implementations aren't "broken", so much as "limited".
>Well, if they don't honour what the documentation says, then they seem 
>to me, and the documentation doesn't say you might get an unexpected
>64 connection limit. No matter.

This is actually a problem for every reactor; handling of the connection 
limit (and there is a connection limit everywhere, even if it's 65535) 
is poorly documented and poorly handled.

Patches which rectify this situation for any reactor, either from the 
perspective of docs or code, would of course be appreciated.
>>It still doesn't support SSL, though:
>>    http://twistedmatrix.com/trac/ticket/593
>Need something patterned very much like Len's stuff
>here http://www.lenholgate.com/archives/000456.html I guess.

The real solution to this problem is really:

  * Add an SSH IProtocol implementation that provides ITransport, rather 
than building it into the reactor
  * Add an API to ITransport to switch protocols, so that the protocol 
doesn't need to support protocol switching

The BIO stuff mentioned on that URL is vaguely correct, but the 
interesting stuff is exposing it to Python.  Now that JP Calderone has 
taken over maintenance of PyOpenSSL, we might actually have a chance to 
get that done :).

Once you've got an SSL connection that is getting bytes from and 
delivering bytes to a transport, you don't need to care about any of the 
win32-specific aspects of that post; you just do the I/O using whatever 
platform I/O facility you want and the crypto happens completely 
disconnected from this socket.  iocpreactor already implements enough of 
this to work.

Again, if you want to see this happen: write code which does these 
things and contribute it.
>>Your suggestion for handle buckets in the win32eventreactor is one 
>>which has been suggested before and sounds good to me.  Nobody has 
>>bothered to implement it yet :).  However, even in its absence, I 
>>don't believe we'll be completely dropping win32eventreactor because 
>>it is useful for integrating with programs that do win32 drawing, even 
>>if iocpreactor has higher performance and is more scalable.
>Well, you need a daemon worker thread to call 

(... blah blah win32 API stuff that I don't understand ...)
>Why do you need an event reactor to support win32 applications, except 
>to call MsgWait... rather than GetQueuedCompletionStatus?

My answer here, if I even have one, would be horribly vague: something 
about GDI+, or DirectX, or something.  Maybe there's no good reason, I 
don't know.  This level of specificity is probably best served by the 
issue tracker; file a ticket.  And, if you're going to file a ticket, be 
prepared to actually follow up with an implementation.  While we want to 
maintain support for Windows, the level of energy for doing really 
interesting Windows-specific stuff in Twisted is, in a word, "low".

One thing you might want to know before you file that ticket is that 
*if* there's a reason for the way things are now, it's because in the 
Twisted idiom we always make sure GUI stuff and network stuff is 
happening in the same thread.  If one of the approaches that you 
mentioned requires a dedicated "network" application thread, then that's 
probably why we aren't doing it.
>On a similar sort of topic - is there a reason to have lots of 
>implementations for POSIX, rather than use libevent or libev?

There is more than one answer to this question.  Maybe someone will be 
helpful and turn some of these answers into a FAQ on the wiki:

1) Twisted predates libevent by a few years and libev by many years. 
One might instead ask why libevent didn't help us develop a C reactor, 
rather than writing a whole new library.

2) There's a libevent reactor in a branch.  Contribute patches to deal 
with the outstanding review comments: 

3) Twisted runs on many platforms, so it supports the lowest common 
denominator; that is not libevent, it is Python's "select" module.

4) Neither libevent nor libev (to my knowledge) integrates with any GUI 
event loops.

5) Despite many valid rationalizations for its existence, the code in 
Twisted was developed organically over many years.  The stuff you'll 
find here is the stuff that people thought was interesting and had time 
to work on.  Strategically standardizing on a single low-level 
multiplexing mechanism is not something that is particularly fun or 
rewarding, especially when getting rid of the old code removes value for 
some users.  (Not everyone already has libevent installed.)

More information about the Twisted-Python mailing list