[Twisted-web] Question regarding custom contextFactory for twisted.web.client.Agent

[Glyph]

> On May 13, 2017, at 7:36 PM, Jason Litzinger <jlitzingerdev at gmail.com> wrote:
> 
>> This sounds eminently sensible, and in fact sounds like Twisted, or Treq at least, really ought to do something like this on our end with some kind of small LRU cache, keeping the most-used N hostnames (where N defaults to some small number, say, 20).  I'm surprised to hear it is such a big optimization, but surprises like that are entirely the point of performance testing!
> Thanks for taking a look!
> 
> I like the idea of an LRU for the last N hosts.  Given I have something
> prototyped that I will expand on, that isn't much of a stretch.  If
> you think it might be useful to Twisted then I can try to iterate the
> patch until it is either rejected or accepted (following the well
> documented steps for contributing of course), thoughts?

This sounds like exactly what you should do.  (Keeping in mind that very nearly 100% of patches are "rejected" on their first attempt, so please do resubmit :))

> If it fits more in treq then I'm more than happy to share the code, but,
> since I'm not currently using treq the amount of effort I can
> realistically put towards it isn't as high, though that project is very
> appealing.

Probably the best place to put it would be Twisted proper.  I just mentioned Treq because sometimes utility functionality like this can become surprisingly complex and have a lot of dependencies, and pin Treq it's easier to make the case that it's a high-level API that should just always do the right thing, as opposed to Agent which needs to preserve maximum flexibility for all HTTP client applications.

> On the size of the performance increase -- I was surprised as well, and
> there's always the possibility I made an error, but it was consistent
> across the five tests I ran.  Regardless, I'll be doing more profiling as
> I work toward the final implementation.

Like I said, if we were never surprised we wouldn't have to have tests :-).

>> Nevertheless, thanks for asking!  I really wish people would ask questions about client TLS more often before doing something potentially insecure :).  In this case, the client connection creator is used to produce client connections anyway, and the connections are not shared, so there should be no worrisome mingling of state between connections.
> Agree.  I've come to believe that assumptions mixed with TLS/security
> related issues are my worst enemy.

Not just yours; the latin for "HTTP-only hist" is "hostis humani generis".

> -Jason
> 
> * Completely unrelated - I still owe on my committment to ticket
>  6597...yeah, life happened.  Luckily I saved my notes now that I'm
>  finding some time here and there.

We do what we must, because we can.

-glyph