[Twisted-Python] sharing a dict between child processes

Maarten ter Huurne maarten at treewalker.org
Wed Nov 6 07:21:22 MST 2019


On Wednesday, 6 November 2019 07:19:56 CET Waqar Khan wrote:
> Hi,
> So, I am writing a twisted server. This server spawn multiple child
> processes using reactor spawnProcess that initializes a process
> protocol.
> 
> Now, each of the childprocess receives some REST requests. Each
> process has a dict that acts as cache.
> Now, I want to share dict across processes.
> In general, python has SharedMemoryManager in multiprocessing module
> which would have helped.
> https://docs.python.org/3/library/multiprocessing.shared_memory.html#m
> ultiprocessing.managers.SharedMemoryManager.SharedMemory But since I
> am using twisted internal process implementation, how do I share this
> dict across the processes so that all the processes use this common
> cache?

Keeping a dictionary in SharedMemoryManager seems far from trivial. I 
don't think you can allocate arbitrary Python objects in the shared 
memory and even if you could, you would run into problems when one 
process mutates the dictionary while another is looking up something or 
also mutating it.

It could in theory work if you implement a custom lock-less dictionary, 
but that would be a lot of work and hard to get right. Also having 
shared memory mutations be synced between multiple CPU cores could 
degrade performance, since keeping core-local CPU caches in sync is 
expensive.

Would it be an option to have only one process accept the REST requests, 
check whether the result is in the cache and only distribute work to the 
other processes if you get a cache miss? Typically the case where an 
answer is cached is pretty fast, so perhaps you don't need multiple 
processes to handle incoming requests.

Bye,
		Maarten






More information about the Twisted-Python mailing list