[Twisted-Python] Running CPU bound function concurrently in twisted

Chengi Liu chengi.liu.86 at gmail.com
Mon Jun 24 18:24:00 MDT 2019


I think I got around the second problem (thanks for sharing the react
documentation)

How do I use more multiple CPU cores? I guess atleast the fact that twisted
app wont be blocking is good, but what would be a good way to use multiple
cores.

I do see that meejah suggested using multiple python processes while
setting up the reactor?

My question is that.. say.. I set up multiple twisted based python
processes by leveraging the spawnProcess..

And then we setup a resource 'foo'.. When the request is hit to `foo` end
point... will the parallelization happen here..
say.. I have spawned 2 processes and one request came and hit the 'foo' end
point.. I am guessing.. the fact that 2 processes is running would help
responding to multiple requests simultaneously.

I think that level of parallelization is good and I will take it.. But
here.. what I want is... when you hit "foo".. then there is a function that
is parallelizable..
Like.. you hit "foo" endpoint.. I fetch data from somewhere.. (yay treq)..
and then there is a function that is called when we hit "foo" end point and
that function is extremely parallelizable over for loop. Now, what is great
is that with all your help, I am able to make this flow non-blocking. But,
would definitely like to exploit parallellism wherever feasible.



On Mon, Jun 24, 2019 at 7:37 PM Moshe Zadka <moshez at zadka.club> wrote:

> On Mon, Jun 24, 2019, at 16:27, Chengi Liu wrote:
>
>      cpu_res *= yield *defer.*gatherResults*(cpus)
>
>
> Remember: This will not block the reactor (Good!) but will still limit you
> to one CPU core (in general, some caveats, but true.) If you are CPU bound,
> this is woefully underutilizing modern CPU resources (again, some caveats
> here).
>
> if __name__ == '__main__':
>
>
> In general, the best thing to do is to to use twisted.internet.task.react
> for main. This would also solve the other problem you point out below.
>
>
> https://twistedmatrix.com/documents/current/api/twisted.internet.task.html#react has
> a good example.
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20190624/f5486a90/attachment-0002.html>


More information about the Twisted-Python mailing list