[Twisted-Python] Synchronization techniques
glyph at divmod.com
glyph at divmod.com
Wed Apr 4 13:43:02 EDT 2007
On 04:35 pm, daniel at keystonewood.com wrote:
>On Apr 3, 2007, at 8:42 PM, Itamar Shtull-Trauring wrote:
>>On Tue, 2007-04-03 at 17:07 -0400, Daniel Miller wrote:
>>>>twisted.internet.defer.DeferredLock and some of the related classes
>>>>what you ought to be using.
>>>Unfortunately that only gets me half way there. DeferredLock.acquire
>>>() returns a deferred. How do I return the result of a deferred from
>>>a PB remote_xxx() function?
>>Just return the Deferred from the remote_xxx() function.
>Thanks, I didn't know I could return a deferred from a PB remote_xxx ()
>method. That detail doesn't seem to be documented in the Perspective
>Broker documentation, which I have read quite a few times.
The PB documentation is not too great. Perhaps this paper would be
helpful to you, if you haven't seen it:
In addition, the remote method can itself return a Deferred instead
actual return value. When that Deferreds fires, the data given to the
callback will be serialized and returned to the original caller.
>Maybe this could be highlighted in the "Complete Example"  section
>of the PB usage documentation? The examples use the TwistedQuotes
>application, and the IQuoter.getQuote() method always returns a string
>(at least I couldn't find any implementations that return a deferred).
Please feel free to write some patches for the documentation, or open a
doc bug describing this issue in more detail. It's definitely an under-
documented feature of PB.
>However, that would require rewriting most if not all implementations
>of IQuoter to return deferred's and/or the code that calls
>IQuoter.getQuote(), which demonstrates the viral nature of twisted
>when used in conjunction with other libraries.
I don't think that would really be the terrible burden that you suggest,
considering the relatively small amount of tutorial documentation that
implements or calls IQuoter. One could also propose a separate
interface, IDeferredQuoter, to make the distinction clearer.
>So anyway, I rewrote my server-side library to do it the twisted way
>and return deferred's instead trying rig up some way of waiting for
>them. I still think it would be super useful to be able to pseudo-
>block on a deferred (i.e. allow the reactor to process other events
>while waiting for the deferred). It is very annoying to have to
>rewrite many layers of code when twisted is introduced into a program.
>I did find gthreadless.py, and maybe that would do it. Unfortunately
>discussion on that seems to have been dropped some time ago...
I'm afraid that the feature you want doesn't make any sense and is, in a
broad sense, impossible. There are some things like it which might be
possible - for example, http://twistedmatrix.com/trac/ticket/2545 - but
the reactor is not reentrant and in some sense could not be made
Consider this innocuous looking block of code:
from twisted.internet.protocol import Protocol
from make_believe import magicallyBlockOn
def dataReceived(self, data):
commands = (self.buf + data).split()
self.buf = commands[-1]
for command in commands[:-1]:
if command == 'QUIT':
# Deferreds are hard, let's go shopping
If you were using Deferreds to track the result of the 'getPage'
operation, you could cancel the callbacks that write to the transport in
connectionLost. However, with magical blocking, one dataReceived method
might be interleaved with another. That means that every time through
the loop, you have to check to see if the transport has already been
disconnected - the code as presented here is buggy and will spuriously
fail depending on the order of the connection being lost and the remote
page being retrieved.
In this example I've been careful to accumulate all the buffer-
management and parsing logic at the top of the method, before any
potential re-entrancy can happen, but other code (most everything in
Twisted's existing protocol implementations, not to mention just about
all application code) would not be so lucky.
It might be quite feasible to implement a microthreaded runtime
environment that lived on _top_ of Twisted and explicitly accounted for
issues like these, but that would not really be substantially different
>For the record, I've included updated versions of the previously
>posted code below. I'd be happy if someone pointed out if I'm doing
>anything wrong (with respect to twisted) in this code.
Nothing immediately jumps out at me. I've had to write similar code in
the past, though, and when I've had to do that, an explicit state
machine for the state of the subprocess (or whatever asynchronous
resource must be acquired) has been easier to deal with than a lock-
oriented approach to it.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python