[Twisted-web] Question regarding custom contextFactory for twisted.web.client.Agent

[Glyph]

> On May 12, 2017, at 1:56 PM, Jason Litzinger <jlitzingerdev at gmail.com> wrote:
> 
> Hello,
> 
> I have an optimization question in the realm of "before I do something
> foolish..." regarding the contextFactory passed to twisted.web.client.Agent.
> 
> Background:
> 
> I have (several) twisted applications running on a Cortex-A5 system, one of
> which interfaces with a web service using the twisted.web.client.Agent
> API*.  The
> problem I've observed is that requests to this web service result in a higher
> than desired CPU load.  To understand the source of the load I isolated the
> server interactions and profiled.
> 
> Investigation:
> 
> The profile data revealed the load in question stemmed from TLS related
> calls, specifically optionsForClientTLS, which is called once per connection.
> In looking (briefly) at what optionsForClientTLS does, it seemed
> (perhaps wrongly)
> that there was an opportunity for optimization in my specific case.
> 
> The optimization is to create a custom contextFactory that caches the connection
> creator since the host is not changing.  I prototyped an implementation and the
> load was reduced by a factor of ten.
> 
> Question:
> 
> Is it safe/sane to reuse a connection creator if the host to which I'm
> connecting is
> not going to change?

This sounds eminently sensible, and in fact sounds like Twisted, or Treq at least, really ought to do something like this on our end with some kind of small LRU cache, keeping the most-used N hostnames (where N defaults to some small number, say, 20).  I'm surprised to hear it is such a big optimization, but surprises like that are entirely the point of performance testing!

Nevertheless, thanks for asking!  I really wish people would ask questions about client TLS more often before doing something potentially insecure :).  In this case, the client connection creator is used to produce client connections anyway, and the connections are not shared, so there should be no worrisome mingling of state between connections.

We do host a performance test suite here: http://speed.twistedmatrix.com <http://speed.twistedmatrix.com/> and the benchmarks are here https://github.com/twisted-infra/twisted-benchmarks <https://github.com/twisted-infra/twisted-benchmarks> if your test is not adequately represented in an existing benchmark.

-g

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-web/attachments/20170512/ac64aab2/attachment.html>