Introduction
I would like to use Twisted in Zope Enterprise Objects (ZEO) which is
the client-server component of the Zope Object Database (ZODB). After
some discussion on the #twisted IRC channel and some follow-on private
e-mail, I'll classify ZEO as a non-Twisted library that uses Twisted.
It is a non-Twisted library because it provides operations that may
block and are therefore inappropriate for use by code invoked directly
from a Twisted reactor.
There are two issues which may potentially affect such libraries. First,
such a library may need a reactor to be running. For example, to read
data using ZEO, the reactor used by ZEO has to be running. Consider
two cases:
a. ZEO is used in an application that doesn't use Twisted.
In this case, the host application won't start a reactor. The use
of Twisted is an internal implementation detail of ZEO, so it
should be ZEO's responsibility to start the reactor.
b. ZEO is used in an application that uses Twisted. (Note that this
application may not be a Twisted application. For example, a web
application that uses the Twisted Web 2 WSGI server is not a
Twisted application.) The application will need to start the
reactor to function. Normally, this is done at the end of the
application's main entry point. If the application uses ZEO, this
is likely to be too late. The application is likely to need to
read database data during start up. Coordinating reactor management
is problematic, at best, in a situation like this.
The second issue has to do with use of non-Twisted libraries that use
Twisted in Twisted applications. For example, consider a Twisted
application that requests data from ZEO while responding to a call
from a reactor. If the needed data isn't available locally, then ZEO
will need to download the data from the server. If ZEO uses the same
reactor as the application, then it will block waiting for data that
will never arrive because the reactor is blocked waiting for the
return of the application code that requested data from ZEO and
deadlock will occur. The Twisted application in this scenario is
broken. The deadlock described here should not occur,
because code called directly from a reactor should only invoke ZEO in
a separate thread, however, the risk exists that someone will
inadvertently call ZEO when they shouldn't, and, given the severity of
deadlock, the risk seems worth mitigating. If ZEO had it's own
reactor, there would be no deadlock in this scenario.
An apparent obstacle to having multiple reactors is the use of the
twisted.internet.reactor global. Most code can avoid this, however,
because reactors are available via transport reactor attributes. For
example, a protocol used with a specialized reactor can access the
reactor via self.transport.reactor. I'm using this mechanism now to
allow use of a test reactor in a protocol I'm working on.
Another obstacle is the use of module globals in the SelectReactor
implementation.
Minimal Proposal
I propose to:
1. Add the reactor attribute to ITransport.
This will make it acceptable to use the attribute and will,
hopefully, encourage use of the attribute rather than the global
variable.
2. Add the ability to create multiple SelectReactors.
It appears that this will entail: moving the reads and writes
global dictionaries to instance variables.
With these changes, I believe that non-Twisted libraries will be able
to manage and use their own reactors. When a library is imported, it
will set up it's own internal reactor with code along the following
lines:
import threading
import twisted.internet.selectreactor
reactor = twisted.internet.selectreactor.SelectReactor()
thread = threading.Thread(target=reactor.run, args=(False,))
thread.setDaemon(True)
thread.start()
It will then register servers or request connections by using the
reactor's callFromThread method to call listenXXX or connectXXX
methods.
I believe that this change will meet my needs and the needs of others
who want to create non-Twisted libraries that use Twisted.
I volunteer to do this work.
Grand Proposal
Existing code that uses `twisted.internet.reactor' could be converted
to use local reactors. This can be easily done for code, such as
protocol implementations, that has access to transports.
I think this would provide a certain level of cleanliness and would
make it easier to reuse existing software with private reactors,
including test reactors.
I'd be happy to at least help with this, especially when the changes
are mechanical and, therefore, easy. :)
