[Twisted-Python] doWrite on twisted.internet.tcp.Port

[Jean-Paul Calderone]

On Fri, Sep 11, 2020 at 1:34 PM <chris at cmsconstruct.com> wrote:

> Hey guys,
>
>
>
> Last year I hit a condition discussed in this ticket:
> https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a
> twisted.internet.tcp.Port.
>
>
>
> I ignored it at the time since it was just on Linux, and my main platform
> was Windows.  Now I’m coming back to it.  I’ll add context on the problem
> below, but first I want to ask a high-level, design-type question with
> multiprocessing and Twisted:
>
>
>
> Referencing Jean-Paul’s comment at the end of ticket 4759, I read you
> shouldn’t fork a process (multiprocessing module) that already has a
> Twisted reactor.  Understood.  But what about a parent process (not doing
> anything Twisted) forking child processes, where each child process starts
> their own Twisted reactor?  Is that intended to work from the Twisted
> perspective?
>

To answer the asked question, I don't think there is rigorous (or even
casual) testing of very much of Twisted in the context of "some Twisted
code has been loaded into memory and then the process forked".  So while it
seems like a reasonable thing, I wouldn't say there's currently much effort
being put towards making it a *supported* usage of Twisted.  Of course this
can change at almost any moment if someone decides to commit the effort.

To dig a bit further into the specific problem, even if you only *import* the
reactor in the parent process and then fork a child and try to start the
reactor in the child, I strongly suspect epollreactor will break.  This is
because the epoll object is created by reactor instantiation (as opposed to
being delayed until the reactor is run).  epoll objects have a lot of weird
behavior.  See the *Questions and Answers* section of the epoll(7) man page.

I don't know if this is the cause of your particular expression of these
symptoms (it certainly doesn't apply to the *original* bug report which is
on FreeBSD where there is no epoll) but it's at least *a possible* cause.

This could probably be fixed in Twisted by only creating the epoll object
when run is called.  There's nothing particularly difficult about that
change but it does involve touching a lot of the book-keeping logic since
that all assumes it can register file descriptors before the reactor is
started (think reactor.listenTCP(...); reactor.run()).

I'm not sure but it may also be the case that only delaying creation of the
*waker* until the reactor starts would also fix this.  This is because as
long as the epoll object remains empty a lot of the weird behavior is
avoided and the waker is probably the only thing that actually gets added
to it if you're just *importing* the reactor but not *running* it before
forking.

Alternatively, your application *should* be able to fix it by studiously
avoiding the import of twisted.internet.reactor (directly or transitively,
of course).  You could add some kind of assertion about the state of
sys.modules immediately before your forking code to develop some confidence
you've managed this.

And if this is really an epoll problem then switching to poll or select
reactor would also presumably get rid of the issue.

Jean-Paul


>
>
>
> Context:
>
> I only hit this problem on Linux, not Windows.
>
>
>
> The software project (github.com/opencontentplatform/ocp) has been a lot
> of fun, especially with walking the tight rope in using multi-processing,
> multi-threading, and Twisted reactors.  The main controller process kicks
> off about 10 child processes, each doing different types of work.  In
> general though, the child processes individually start a Twisted reactor,
> connect to Kafka, connect to a database, use a shared REST API, and some
> listen for connecting clients to accomplish work.
>
>
>
> I test on Linux about once a year, so too many changes to rollback and
> figure out that way.  It was working on Linux 2 years ago, but last year’s
> testing and current testing, receive the doWrite error.  It continues
> running fine on Windows.  I’ve gone back about 2 years of versions with
> Python3, Twisted, and dependent libs… on both Windows and Linux.  Every
> version change yields the same result - continues to work on Windows and
> continues to hit the error on Linux.  So something I added has caused Linux
> to throw that error.
>
>
>
> I’m not explicitly sharing much between the main process and the sub
> processes.  They are spun up with (1) a shared multiprocessing.Event() - to
> have the main process shut the children down, (2) their own unique
> multiprocessing.Event() - to have the child processes notify back to the
> parent, and (3) a deep copy of a dictionary (containing a bunch of settings
> that remain constant).  The main process uses twisted.logger, but for
> testing I strip that out to remove any twisted imports in the main
> process.  So I’m not importing anything Twisted in the main process, and I
> don’t believe I’m explicitly sharing something I shouldn’t.  Seems like
> something is implicitly being exposed/shared across Linux child processes,
> that aren’t on Windows.
>
>
>
> The tracebacks come through on Linux (sometimes randomly), on the console
> of the parent controller process.  No need to paste here, since it’s the
> same as the ticket shows.  I can’t reliably reproduce the problem, but I
> know if I stop/start client connections
> (ServerFactory/twisted.internet.interfaces.IReactorTCP and
> twisted.internet.protocol.ReconnectingClientFactory) then it will
> eventually happen.  I need to devote time at whittling down the code and
> attempting to create a reliable test case… if even possible.
>
>
>
> The error is slightly different when running HTTP vs HTTPS, but the story
> is the same.  It cripples whatever child process that hits it, from doing
> much of anything thereafter.  Not much luck with troubleshooting.  The
> tracebacks do not include a calling function from my code, to tell me where
> to start. And it happens across different child process types, so not the
> same one each time.  When I throw debuggers on the child processes, the
> problem seems to mask itself.  Well, at least I didn’t hit the problem over
> the last 3 days using pudb and stepping through code at breakpoints.
>
>
>
> I’m absolutely open to suggestions for troubleshooting, but first wanted
> to take a HUGE step back and ask a design question regarding Twisted and
> multiprocessing.
>
>
>
> Thanks!
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20200911/e69a2ccc/attachment.htm>