[Twisted-Python] Twisted tips for designing highly concurrent twisted REST API
Waqar Khan
wk80333 at gmail.com
Wed Jun 26 00:50:57 MDT 2019
Sorry I had a typo in twisted program
@defer.inlinecallbacks
def long_computation(rec_type, data):
# some long computation
*defer.returnValue(recs)*
@defer.inlinecallbacks
def fetch_data(user_id):
r = yieldjson.loads(requests.get('url/to/fetch/%s'%user_id).text)
defer.returnValue(r)
@defer.inlinecallbacks
def fetch_recs(user_id):
data = yield fetch_data(user_id)
recs = {}
for stype in similar_types:
*d = defer.ToThread(long_computation, *(stype, data)) // typo
was here*
rec = yield d
recs[stype] = rec
defer.returnValue(recs)
On Tue, Jun 25, 2019 at 11:48 PM Waqar Khan <wk80333 at gmail.com> wrote:
> Hello folks,
> I recently stumbled upon twisted and was wondering if it could suit my
> needs. On one hand, I want to use python but on another hand there are all
> these scalability concerns with this language so, I though I would pick the
> brains of the community. So.. a flask based app would look something like
> this.
>
> similar_types = ['foo', 'bar', 'baz']
>
>
> def long_computation(rec_type):
> # some long computation
> return recs
>
> @app.route('/fetch_similar_users/<user_id>'
> def fetch_similar_users(user_id)
> r = json.loads(requests.get('url/to/fetch/%s'%user_id).text)
> recs = {}
> for stype in similar_types:
> recs[stype] = long_computation(rec_type)
> return recs
>
>
>
> Now, I tried to "twistify" but it failed.
>
> @defer.inlinecallbacks
>
> def long_computation(rec_type):
> # some long computation
> *defer.returnValue(recs)*
>
> @defer.inlinecallbacks
> def fetch_data(user_id):
> r = yieldjson.loads(requests.get('url/to/fetch/%s'%user_id).text)
> defer.returnValue(r)
>
>
> @defer.inlinecallbacks
> def fetch_recs(user_id):
> data = yield fetch_data(user_id)
> recs = {}
> for stype in similar_types:
> d = defer.ToThread(fetch_data, *(stype))
> rec = yield d
> recs[stype] = rec
> defer.returnValue(recs)
>
>
>
> I wrapped all the above in twisted render_Get method.. but then I did a
> load test with locust (
> https://docs.locust.io/en/latest/what-is-locust.html) framework.
> It choked. As the time progressed, the response time increased.
> I am guessing, things are still blocking.
>
> Can you please help me look into the right place. Why exactly am I seeing
> increase in response time as the time progresses. I am guessing things are
> still working in "blocking" fashion but i thought the above should run
> things in async.
> Thanks
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20190625/608e78e2/attachment-0002.html>
More information about the Twisted-Python
mailing list