[Twisted-Python] Twisted tips for designing highly concurrent twisted REST API

[Waqar Khan]

Hi,
  Thank you all for your kind response.

So, I am trying to use treq library

import treq

@defer.inlinecallbacks

def long_computation(rec_type, data):
     # some long computation
     *defer.returnValue(recs)*

@defer.inlinecallbacks
def fetch_data(user_id):
        r = yield treq.get('url/to/fetch/%s'%user_id)
    text = yield r.text()
     defer.returnValue(text)


@defer.inlinecallbacks
def fetch_recs(user_id):
       data = yield fetch_data(user_id)
        recs = {}
       for stype in similar_types:
            *d = defer.ToThread(long_computation, *(stype, data)) // typo
was here*

Now, I do believe that the call is happening asyncronously. So.. yay..

But then, I feel like I have a misconception on how the yield works.
data = yield fetch_data(user_id)
I was hoping data here was actual data.. But it is a deferred.. Which makes
sense.
And then.. this deferred is being passed on instead of the actual data...

My couple of questions are:
1) What is the difference between data = yield fetch_data(user_id) and data
= fetch_data(user_id) (without yield). How does twisted handle these two ?
2) How do I actually send the data to long computation rather than a
deferred.

Appreciate all the help.
Thanks


On Sat, Jul 13, 2019 at 1:57 AM Tom Most <twm at freecog.net> wrote:

> On Thu, Jul 11, 2019, at 1:46 AM, Scott, Barry wrote:
> > On Tuesday, 9 July 2019 22:04:11 BST Tom Most wrote:
> >
> > ...snip...
> >
> > > The reactor's own thread pool is really for DNS resolution.
> >
> > Is that still true in the default case? We are use the twisted code that
> talks
> > to DNS servers as the threaded resolver adds too much latency.
>
> As far as I know, yes. The higher-level APIs use getaddrinfo() at least.
>
>
> https://twistedmatrix.com/documents/current/api/twisted.internet._resolver.GAIResolver.html
>
> https://github.com/twisted/twisted/blob/c0776850e756adfcdc179a7fd9e4c8f5cbc4838d/src/twisted/internet/base.py#L1007
>
> TCP6ClientEndpoint also invoke getaddrinfo() directly.
>
> twisted.names is certainly more performance but it's missing some system
> integration features that make it unsuitable as a default:
>
> * No support for the domain or search resolv.conf directives
> * No NSS lookups (e.g., systemd integration)
>
> This is all on Linux, YMMV on other platforms.
>
> ---Tom
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20190713/7a922c7d/attachment-0002.html>