[Twisted-web] How does Twisted Web Multiprocess work ?

[Gavin Panella]

On 10 June 2015 at 23:39, Sagar Dixit <sagar.dixit at gmail.com> wrote:
> Thanks Gavin
>
> Good to know how things are done at your end. thanks! However, why is
> Twisted not internally using SO_REUSEPORT ? Is there any plan to
> integrate it inside twisted so that application need not worry about
> it ?

I suspect it's only because it's fairly new in Linux and support hasn't
been written/merged into Twisted. Someone else might have detail to add
to that; in this regard I'm only a user of Twisted (for now... maybe
I'll get an itch to address this in Twisted itself if someone else
hasn't already done so).

> The problem with code mentioned here:
> http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multiprocessor
> is that the requests are not equally distributed across all processes.
> This imbalance leads to underutilization of CPU cores. This issue is
> also discussed on https://lwn.net/Articles/542629/ and that this is
> not an issue with SO_REUSEPORT

That's interesting. We used SO_REUSEPORT for convenience but it's good
to know that it's balancing nicely too :)

> Does it mean that Twisted is using single listening socket and all
> processes accept() on that socket ?

Each process gets its own socket, but the kernel allows all of them to
bind to the same local address:port.

If you need more processes you can start them without ceremony and it'll
all Just Work. Likewise when killing processes. Done right, you can have
seamless upgrades of your service by restarting one process at a time.
No need for special support from a supervisor.

Gavin.