[Twisted-Python] sharing a dict between child processes

[Scott, Barry]

On Wednesday, 6 November 2019 14:21:22 GMT Maarten ter Huurne wrote:
> On Wednesday, 6 November 2019 07:19:56 CET Waqar Khan wrote:
> > Hi,
> > So, I am writing a twisted server. This server spawn multiple child
> > processes using reactor spawnProcess that initializes a process
> > protocol.
> > 
> > Now, each of the childprocess receives some REST requests. Each
> > process has a dict that acts as cache.
> > Now, I want to share dict across processes.
> > In general, python has SharedMemoryManager in multiprocessing module
> > which would have helped.
> > https://docs.python.org/3/library/multiprocessing.shared_memory.html#m
> > ultiprocessing.managers.SharedMemoryManager.SharedMemory But since I
> > am using twisted internal process implementation, how do I share this
> > dict across the processes so that all the processes use this common
> > cache?
> 
> Keeping a dictionary in SharedMemoryManager seems far from trivial. I
> don't think you can allocate arbitrary Python objects in the shared
> memory and even if you could, you would run into problems when one
> process mutates the dictionary while another is looking up something or
> also mutating it.
> 
> It could in theory work if you implement a custom lock-less dictionary,
> but that would be a lot of work and hard to get right. Also having
> shared memory mutations be synced between multiple CPU cores could
> degrade performance, since keeping core-local CPU caches in sync is
> expensive.
> 
> Would it be an option to have only one process accept the REST requests,
> check whether the result is in the cache and only distribute work to the
> other processes if you get a cache miss? Typically the case where an
> answer is cached is pretty fast, so perhaps you don't need multiple
> processes to handle incoming requests.

We have used a couple of ways to cache.
1. Use a singleton process to hold the cache and ask it, via IPC, for answers 
from the other process.
2. have a cache in each process

Barry


> 
> Bye,
> 		Maarten
> 
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python