[Twisted-Python] Twisted and PEP 3148 support

[Tim Allen]

On Mon, Jan 10, 2011 at 11:57:14PM +0200, Alex Grönholm wrote:
> The main focus of the discussion is to figure out how to best integrate 
> support for this new API to Twisted. If possible, existing protocols 
> should remain compatible through the use of adapters or some other 
> mechanism. If not, a way to port them over with a minimal amount of work 
> would be the next target.

I think I've read PEP 3148 before, but that was a while ago. Looking
over it again now, here are my initial thoughts as just a user of
Twisted.

- Executor.submit() seems to be an analogue to Twisted's
  reactor.callInThread(), that should be easy enough to add to Twisted's
  implementation.
- Executor.map() doesn't havve a Twisted analogue, as far as I know, but
  seems a useful addition.
- Executor.shutdown() seems like a Bad Idea, at least for implementation
  in Twisted's reactor - I don't want to call some third-party library
  and have it shut down my entire process when I'm done! Perhaps, rather
  than making Twisted's reactor implement the Executor interface, there
  should be .makeExecutor() method that returns an Executor instance
  that can support the .shutdown() semantics without actually shutting
  down the Twisted event loop.

- Futures are obviously fairly similar to Deferreds, but there are some
  differences of approach:
    - Futures support cancellation; Twisted finally managed to get rid
      of cancellation support in Deferreds.
    - Futures pass themselves to a series of callbacks, and have
      .result() and .exception() methods so that the callbacks can
      discover what happened. Deferreds pass results or Failure objects
      through a double list of callbacks and errbacks.
    - Futures store the results of the function call that was run in
      a thread, and pass the same information to each calback. Deferreds
      allow each callback to transform the result and even pause the
      callback-chain to wait for more asynchronous results.
    - Futures allow code to wait indefinitely for a result or exception
      to appear, which makes sense if the result is being calculated in
      a thread, but which would cause a deadlock in an event-based
      system like Twisted.

It should be pretty simple to create a Deferred that wraps a Future:

    from twisted.internet import defer

    def deferredFromFuture(future):
	d = defer.Deferred()
	def callback(future):
	    e = future.exception()
	    if e:
		d.fail(e)
		return

	    d.succeed(future.result())

	future.add_done_callback(callback)
	return d

I guess a creating a future that wraps a Deferred wouldn't be hard
either:

    from concurrent.futures import Future

    def futureFromDeferred(deferred):
	f = Future()
	f.set_running_or_notify_cancel()

	def callback(result):
	    f.set_result(result)
	    return result
	d.addcallback(callback)

	def errback(failure):
	    f.set_exception(failure.value)
	    return failure
	d.adderrback(errback)

	return f

Of course, if somebody added another callback after the ones added by
futureFromDeferred(), the new values wouldn't be passed to the Future,
and if somebody called future.result() or future.exception() without
a timeout, that could cause a deadlock as mentioned above. I'm not sure
it's possible to work around either of those problems, so perhaps
futureFromDeferred() is a thing that should not be included in
a library, at least not without big warnings all over it like Twisted's
integration with the stdlib "logging" module has.

I also can't immediately see a way to make an object that functions as
both a Deferred and a Future, or build Deferred's functionality on top
of Futures. Perhaps I just haven't thought it through sufficiently - I'd
be interested to hear if anyone else has any ideas.