[Twisted-web] How does Twisted Web Multiprocess work ?

[Gavin Panella]

On 10 June 2015 at 03:01, Sagar Dixit <sagar.dixit at gmail.com> wrote:
> Hi,
>
> I am exploring Twisted Web for my RESTful application. My application
> is stateless and involves storing and retrieving objects based on
> Object-ID. This application will run on beefy (multicore, lots of
> memory) machine. However, not all APIs that the application issues to
> underlying storage are async and hence I cannot fully utilize
> Deferreds
> Which means, there will some blocking calls and hence my primary
> interest is to use Twisted Web in multiprocessing mode
>
> I came across
> http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multiprocessor
>
> However, I am not sure if it is the "correct" way of doing things.
>
> Hence I had some questions around it:
>
> 1. Is there an interface (similar to defertoThread) which allows me to
> execute a blocking call in a separate process ?
>
> 2. Does reactor synchronize access of all processes to the shared
> listen socket ?
>
> 3. Is there a sample code I can refer to where the application is
> spawning subprocesses to handle HTTP requests ?

We've used SO_REUSEPORT (which is briefly mentioned on the SO page you
linked to) in order to run multiple processes. On a recentish Linux the
following approach has worked well for us:

    # Make a socket with SO_REUSEPORT set so that we can run multiple web
    # applications. This is easier to do from outside of Twisted as there's
    # not yet official support for setting socket options.
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    # The following might not work on older kernels, or SO_REUSEPORT
    # might not be available if Python was compiled with older headers.
    # In the former case you're out of luck, but you can define the
    # header yourself if it's missing from Python's socket module.
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

    # Listen on all interfaces on port 1234.
    s.bind(('0.0.0.0', 1234))

    # Use a backlog of 50, which seems to be fairly common.
    s.listen(50)

    # Adopt this socket into something more Twisty.
    endpoint = AdoptedStreamServerEndpoint(reactor, s.fileno(), s.family)

    # Prevent garbage collection. This is something we discovered we
    # needed, and is perhaps something that AdoptedStreamServerEndpoint
    # ought to do itself.
    endpoint.socket = s

    # Create a service with the endpoint.
    service = OurService(site_endpoint)

(Derived from src/maasserver/eventloop.py in MAAS.)

We then use Upstart (Ubuntu 14.10 and before) or systemd (Ubuntu 15.04)
to run multiple processes. For us this mean 4, but you can have as many
or as few as you need, for example 1 process per CPU core. I like this
approach because we can use the system's facilities for process
supervision instead of cobbling together our own.

One caveat is that the processes are running concurrently. If you're
modifiying shared resources you'll want to consider explicit locking
where before you might have been safe with Twisted' single-threadedness.

Gavin.