[Twisted-Python] Re-entrancy policy for the reactor

[Glyph Lefkowitz]

On Jul 27, 2011, at 5:41 PM, exarkun at twistedmatrix.com wrote:

>> My main thought here is that protocol reentrancy is bad, and nobody 
>> really expects it even if they think it should be fine.
>> 
>> However, I do believe it would be best (easier to test, in particular) 
>> to immediately call connectionLost within doRead (or doWrite) after 
>> dataReceived exits, rather than callLater(0)-ing it or otherwise 
>> placing it into a global call queue.
> 
> Probably not easier to test, nor easier to get right.  A single 
> generalized solution to avoid re-entrancy is probably the only way to 
> avoid a perpetual sequence of accidental re-entrant calls in obscure 
> untested cases in each different reactor implementation.

Fair point; upon reflection of all the previous ad-hoc non-reentrancy stuff I've seen in the reactor, I retract my suggestion.  For testing, it's easy enough to plug a Clock() in there and iterate it once.

> Perhaps the solution doesn't need to be `reactor.callLater(0, ...)`, but 
> I think various implementations will be better if they put tasks into 
> some kind of queue, perhaps checked at the end of the iteration instead 
> of in the next iteration, rather than special casing each possible re- 
> entrant event in each possible event dispatcher.

What would be the point of adding another task queue to the reactor?  This sounds like a solution to a non-problem.  (Although I might have suggested the same thing myself if I hadn't thought about it too much first.)

The answer that comes to mind when I think about this question is that an end-of-iteration queue would avoid unnecessary additional invocations of whatever multiplexing system call the reactor needs to make.  But there are several points on the down side:

it's more code to maintain
it's more interfaces for users, or at least maintainers, to learn
it's an optimization, and so we should produce a benchmark indicating that the optimization makes sense and is effective before doing it
the optimization could be applied to callLater as it stands; any callLater(0) calls could be executed before the next reactor tick

Mostly though, an additional queue for non-reentrancy would be an ad-hoc special-case solution to one tiny part of the more general problem that reactor event ordering is a complete accident with unforseen consequences.  If there were a more general scheduling mechanism, we could schedule events in a better order overall.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110727/1f81ad96/attachment.htm