[Twisted-web] How does Twisted Web Multiprocess work ?

[Sagar Dixit]

Hi Carl

The only motivation to use multiple processes is to be able to parallelize
blocking calls (since event driven handling is not useful)  In my case
forward proxy is not a bottleneck, however when my code spawns multiple
processes as per
http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multiprocessor

and I run multiple concurrent clients, I see that the server's processes
are not efficiently utilized. Specifically, even if there are outstanding
requests, one of the processes accepts requests lot more while other
processes are idling. I was able to confirm this by printing the Process
IDs of the processes which are handling request

This sort of imbalance could lead to underutilization of CPU - also
discussed here https://lwn.net/Articles/542629/

Hence I was curious to know how Twisted multiprocessing works. It seems
SO_REUSEPORT is not used within Twisted and as per Gavin's reply can be
used "outside" of Twisted.





On Wed, Jun 10, 2015 at 5:42 PM, Carl Waldbieser <cwaldbieser at gmail.com>
wrote:

> There is a twisted developer guide for communicating with child
> processes:
> http://twistedmatrix.com/documents/current/core/howto/process.html
>
> I'm not sure I understand why having multiple processes listen on the same
> socket is desirable in your case.  From reading the articles you linked to,
> it seems like it is only useful in the case where a forward proxy becomes a
> bottleneck.  Is that the case wirh your application?
>
> Thanks,
> Carl Waldbieser
> On Jun 9, 2015 10:03 PM, "Sagar Dixit" <sagar.dixit at gmail.com> wrote:
>
>> Hi,
>>
>> I am exploring Twisted Web for my RESTful application. My application is
>> stateless and involves storing and retrieving objects based on Object-ID.
>> This application will run on beefy (multicore, lots of memory) machine.
>> However, not all APIs that the application issues to underlying storage are
>> async and hence I cannot fully utilize Deferreds
>> Which means, there will some blocking calls and hence my primary interest
>> is to use Twisted Web in multiprocessing mode
>>
>> I came across
>> http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multiprocessor
>>
>> However, I am not sure if it is the "correct" way of doing things.
>>
>> Hence I had some questions around it:
>>
>> 1.  Is there an interface (similar to defertoThread) which allows me to
>> execute a blocking call in a separate process ?
>>
>> 2. Does reactor synchronize access of all processes to the shared listen
>> socket ?
>>
>> 3. Is there a sample code I can refer to  where the application is
>> spawning subprocesses to handle HTTP requests ?
>>
>> Thanks in advance!
>>
>>
>> _______________________________________________
>> Twisted-web mailing list
>> Twisted-web at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>>
>>
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>
>


-- 
ssdixit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-web/attachments/20150610/f65e8641/attachment-0001.html>