[Twisted-Python] Waiting for a contended resource

[Richard van der Hoff]

Thank you for all for all the answers so far, particularly to Ilya and 
Jean-Paul who provided some very helpful code samples.

It's interesting to realise that, by avoiding locking, we can end up 
with a much more efficient implementation. I'll have to figure out how 
widely we can apply this technique - and how often it's going to be 
worth rewriting things to allow that. Thanks for some useful pointers!

Richard


On 12/03/18 20:00, Jean-Paul Calderone wrote:
> On Mon, Mar 12, 2018 at 3:52 PM, Ilya Skriblovsky 
> <ilyaskriblovsky at gmail.com <mailto:ilyaskriblovsky at gmail.com>> wrote:
>
>     Hi, Richard,
>
>     I've used class like this to cache the result of Expensive
>     Calculation:
>
>     class DeferredCache:
>     pending = None
>     result = None
>     failure = None
>
>     def __init__(self, expensive_func):
>       self.expensive_func = expensive_func
>
>     def __call__(self):
>       if self.pending is None:
>           def on_ready(result):
>               self.result = result
>           def on_fail(failure):
>               self.failure = failure
>
>           self.pending =
>     defer.maybeDeferred(self.expensive_func).addCallbacks(on_ready,
>     on_fail)
>
>       return self.pending.addCallback(self._return_result)
>
>
> This seems like basically a correct answer to me. However, I suggest 
> one small change.
>
> You probably want to create and return a new Deferred for each 
> result.  If you don't, then your internal `pending` Deferred is now 
> reachable by application code.
>
> As written, an application might (very, very reasonably):
>
>     d = getResource()
>     d.addCallback(long_async_operation)
>
> Now `pending` has `long_async_operation` as a callback on its chain.  
> This will prevent anyone else from getting a result until 
> `long_async_operation` is done.
>
> You can fix this by:
>
>     result = Deferred()
> self.pending.addCallback(self._return_result).chainDeferred(result)
>     return result
>
> Now the application can only reach `result`.  Nothing they do to 
> `result` will make much difference to `pending` because 
> `chainDeferred` only puts `callback` (and `errback`) onto `pending`'s 
> callback chain.  `callback` and `errback` don't wait on anything.
>
> You have to be a little careful with `chainDeferred` because it 
> doesn't have the recursion-avoidance logic that implicit chaining 
> has.  However, that doesn't matter in this particular case because the 
> chain depth is fixed at two (`pending` and `result`).  The problems 
> only arise if you extend the chain out in this direction without bound.
>
> Jean-Paul
>
>     def _return_result(self, _):
>       return self.failure or self.result
>
>     Using it you can get rid of DeferredLocks:
>
>         deferred_cache = DeferredCache(do_expensive_calculation)
>
>         def getResource():
>             return deferred_cache()
>
>     It will start `expensive_func` on the first call. The second and
>     consequtive calls will return deferreds that resolves with the
>     result when expensive_func is done. If you call it when result is
>     already here, it will return alread-fired deferred.
>
>     Of course, it will require some more work if you need to pass
>     arguments to `expensive_func` and memoize results per arguments
>     values.
>
>     -- ilya
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20180312/7d6bca68/attachment-0002.html>