[Twisted-Python] calling reactor.run() twice in PB app

[Glyph Lefkowitz]

On Wed, 2006-02-15 at 19:59 -0500, Eric Smith wrote:
> Jean-Paul Calderone wrote:
> > On Thu, 16 Feb 2006 10:27:27 +1100, Andrew Bennetts 
> > <andrew-twisted at puzzling.org> wrote:
> >> On Thu, Feb 16, 2006 at 08:17:42AM +1100, Jonathan Lange wrote:
> >>> On 2/16/06, Mike Pelletier <mike at mkp.ca> wrote:
> >>> > What you want is not supported.  If it were, Trial would be a lot 
> >>> simpler.
> >>> > There is some pressure to support it, but I think it's 
> >>> low-priority.  (If I
> >>> > understand correctly.)  Perhaps you can just block the reactor by 
> >>> calling
> >>> > into your app rather than vice-versa?

The easiest thing would be for your program to spawn a Twisted worker
process to get the data, send it along stdout in some simplified form,
and then read its output.

> >>> You understand correctly. It is not supported, Trial would be simpler
> >>> and it is never going to be supported.
> >>
> >> Never is a long time :)
> >>
> >> Last time I pestered glyph about this, I got him to agree that it 
> >> should work.

I think that in that particular conversation I followed up the
reactor-restarting comments with "of course, for truly asynchronous
protocols like PB this is a waste of time, since it doesn't even
conceptually work".

> > Now we just need to find someone to implement it for all the reactors :)
> 
> Well, I happen to be motivated just now.  I'm new to twisted, but I'm 
> not new to Python and I'm reasonably familiar with the style twisted 
> uses (I've written lots of IOCP code).

> If anyone could summarize the problem, and if you're serious about it 
> getting fixed, I could look at it.

If you have the time, energy, and motivation to seriously contribute to
Twisted, AND experience with Win32 and IOCP, it would be a pity to waste
it supporting such an esoteric use-case - 99% of the users who ask for
this are just trying to shoot themselves in the foot, and if they manage
to do it successfully, it's more often bugs in Twisted.  As I said, you
can probably satisfy your own use-case with popen() relatively easily.

Unfortunately this is not a simple problem.  It involves auditing every
aspect of the reactor's internal state, including the threadpool, and
validating that its internal state is consistent before terminating
run().  It also means making sure that every buffer is flushed, all
outstanding bytes are read, etc.  Then you have to do that for each
reactor.  You'd also have to add an API that specifically did this kind
of "soft" shutdown; flushing buffers without closing connections,
stopping all threads, and so forth.

However, it would certainly be a great way to learn the ins and outs of
the reactor API.  I don't particularly want the functionality but I
suspect that adding it will flush out some latent bugs in at least one
reactor implementation, so I'm not opposed to seeing it done.

If you would like to work on something else though, we definitely need a
dedicated Windows maintainer ... ;)