[Twisted-Python] Shared resource manioulation example

[John Santos]

On Fri, 28 May 2010 vitaly at synapticvision.com wrote:

> Hi,
> 
> I'm looking for a Twisted based example how
> it should be properly done update/delete of global dictionary's
> SAME key:value pair from different functions all together.
> Or by another words, two functions are trying to update the same  
> key:value pair of the same dictionary at the same time, than how to do  
> it properly with Twisted?
> 
> Thanks a lot.
> 

Vitaly,

As a relative newbie to both Twisted and Python (I wish I had more time
to play with it, as there are lots of things I don't really understand
yet, but other work intervenes), I *think* the whole point of the
Twisted reactor is to serialize the callbacks.

So two functions *can't* be trying to update the same Python object
(dictionary or anything else, global or local) at the same time.
While one thread has control, it can do anything it wants to the
global dictionary, and no other function can see anything but the
final result.

So "just do it"!

This assumes you aren't using threads, which are *not* compatible
with Twisted anyway.

The only more complicated scenario I can see is if some desired
process uses the current value in a dictionary to initiate a
request to something else (for example, a TCP/IP messages to another
system) which returns its result in another callback which then stores
the result (or some value that depends on the result) in the same
dictionary element.  If two such sequences can overlap at the same
time, but using the initial value of dictionary element in the 2nd
sequence would produce an incorrect final result, but instead it
should be defered until the 1st sequence completes, then you would
need to use some sort of locking mechanism.

A very simple mechanism would be to add a boolean to each object
(i.e. each dictionary value.)  When the first callback executes, it
checks the boolean.  If set, it waits by calling reactor.callLater,
passing itself as the callback.  If the boolean is clear, it sets
it and initiates the processing.  When the final callback executes,
it sets the new value in the dictionary and clears the boolean.

More sophisticated mechanisms could use an exponential waiting
period for the reactor.callLater time; add a timeout (producing
an error if the whole process takes too long, perhaps saving the
initial value so the process could be rolled back if it fails);
replace the callLater's with a different deferal mechanism that
could be called immediately when the lock gets released; check
for deadlocks amoung multiple locks, etc.  There may very well
already exist a Twisted package for doing all this already (complete
with debugging hooks, statistics, etc.)  Someone with more Twisted
experience could probably point you right at it.  If no such
package exists, I think it would be a very nice thing to have.
If people wouldn't mind waiting for a very long (possibly infinite)
time for it, I could probably be coerced into taking a stab at
producing one.


-- 
John Santos
Evans Griffiths & Hart, Inc.
781-861-0670 ext 539