[Twisted-Python] Twisted tips for designing highly concurrent twisted REST API

[Tom Most]

Hi,

There are likely a few things wrong here.

1. You are using requests.get() to make a HTTP request. This is blocking. You might consider using Twisted's Agent <https://twistedmatrix.com/documents/current/api/twisted.web.client.Agent.html> API instead (or treq <https://github.com/twisted/treq>, which puts a requests-like API atop Agent).

2. As you add load your long computations will be queued. deferToThread <https://twistedmatrix.com/documents/current/api/twisted.internet.threads.html#deferToThread> dispatches the long_computation to the reactor's default thread pool <https://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorThreads.html>. This poll has a maximum size and will queue work once it has spun up that many threads.

Rather than using deferToThread (which we should really deprecate as it doesn't accept a reactor parameter...) I'd recommend instantiating your own ThreadPool <https://twistedmatrix.com/documents/current/api/twisted.python.threadpool.ThreadPool.html> and using deferToThreadPool <https://twistedmatrix.com/documents/current/api/twisted.internet.threads.html#deferToThreadPool>. The reactor's own thread pool is really for DNS resolution. You risk deadlocks in a system that ThreadPoolThreadPoolThreadPool

3. The specifics of what long_computation are also important. If it doesn't release the GIL you won't get real parallelism (this is a Python thing, not a Twisted thing). See this recent thread on the topic <https://twistedmatrix.com/pipermail/twisted-python/2019-June/032371.html>.

Though the mechanisms differ athis thread on the topicny of the above would cause the response time to increase as you add load.

Good luck,
Tom

On Tue, Jun 25, 2019, at 11:51 PM, Waqar Khan wrote:
> 
> Sorry I had a typo in twisted program
> 
> *@defer.inlinecallbacks*
> *def **long_computation*(rec_type, data)*:
**     **# some long computation
**     *defer.returnValue(recs)**
> @defer.inlinecallbacks
> def fetch_data(user_id):
> r *= yield*json.*loads*(requests.*get*('url/to/fetch/%s'*%**user_id*).text)
>  defer.returnValue(r)
> 
> 
> @defer.inlinecallbacks
> def fetch_recs(user_id):
>  data = yield fetch_data(user_id)
>  recs = {}
>  for stype in similar_types:
> *d = defer.ToThread(long_computation, *(stype, data)) // typo was here*
>  rec = yield d
>  recs[stype] = rec
>  defer.returnValue(recs)
> 
> 
> On Tue, Jun 25, 2019 at 11:48 PM Waqar Khan <wk80333 at gmail.com> wrote:
>> Hello folks,
>>  I recently stumbled upon twisted and was wondering if it could suit my needs. On one hand, I want to use python but on another hand there are all these scalability concerns with this language so, I though I would pick the brains of the community. So.. a flask based app would look something like this.
>> 
>> similar_types *= *['foo', 'bar', 'baz']
>> 
>> 
>> *def **long_computation*(rec_type)*:
**     **# some long computation
**     **return *recs
>> 
>> *@app.route*('/fetch_similar_users/<user_id>'
>> *def **fetch_similar_users*(*user_id*)
>>         r *= *json.*loads*(requests.*get*('url/to/fetch/%s'*%**user_id*).text)
>>         recs *= *{}
>>         *for *stype *in *similar_types*:
**             *recs[stype] *= **long_computation*(rec_type)
>>        *return *recs
>> 
>> 
>> Now, I tried to "twistify" but it failed.
>> 
>> *@defer.inlinecallbacks*
>> *def **long_computation*(rec_type)*:
**     **# some long computation
**     *defer.returnValue(recs)**
>> @defer.inlinecallbacks
>> def fetch_data(user_id):
>> r *= yield*json.*loads*(requests.*get*('url/to/fetch/%s'*%**user_id*).text)
>>  defer.returnValue(r)
>> 
>> 
>> @defer.inlinecallbacks
>> def fetch_recs(user_id):
>>  data = yield fetch_data(user_id)
>>  recs = {}
>>  for stype in similar_types:
>>  d = defer.ToThread(fetch_data, *(stype))
>>  rec = yield d
>>  recs[stype] = rec
>>  defer.returnValue(recs)
>> 
>> 
>> 
>> I wrapped all the above in twisted render_Get method.. but then I did a load test with locust (https://docs.locust.io/en/latest/what-is-locust.html) framework.
>> It choked. As the time progressed, the response time increased.
>> I am guessing, things are still blocking. 
>> 
>> Can you please help me look into the right place. Why exactly am I seeing increase in response time as the time progresses. I am guessing things are still working in "blocking" fashion but i thought the above should run things in async.
>> Thanks
>> 
>> 
>> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20190709/4bab79b0/attachment.html>