[Twisted-Python] Twisted receiving buffers swamped?

[Tobias Oberstein]

> > Probably the test machine was changed, a new version of OpenSSL or
> pyOpenSSL, or something else?
> 
> One of those things.  There is no infrastructure in place for identifying events
> which impact the performance testing infrastructure.  The only performance

Yes, this is an important point: track changes in infrastructure (everything that might
have an influence, but is outside the tested code).

> testing environment is a very old mac mini still running Snow Leopard, which

omg;)

> > I'd say: the infrastructure aspects when doing performance tests do
> matter. To the degree that performance results are of very limited value at
> all, if the former aspects are not accounted for.
> 
> I don't think the results that we have presently are worth much at all.  My
> point was mostly that there is some infrastructure which is halfway usable,
> and so you don't have to start from scratch.  If you could take over this

You mean taking over the code "as is"

http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/files

or the task in general (Twisted benchmarking)?

> project (I am pretty sure at this point there is nobody to take it over *from*,

We are currently developing performance test infrastructure for Crossbar.io - naturally,
it is eating it's own dog food: the infrastructure is based on Crossbar.io and WAMP to
orchestrate and wire up things in a distributed test setup.

We could extend that to test at the Twisted(-only) level. Need to think about how
that fits into "overall strategy", as the Crossbar.io perf. test stuff isn't open-source.

The testing hardware above (mac, no real network) is insufficient for what I need.
I'm thinking about buying and setting up 2 more boxes for Linux.

Rgd. Codespeed (https://github.com/tobami/codespeed), which seems to be used
by speedcenter.twistedmatrix.com: I have issues here as well.

E.g. I need latency histograms, but this seems unsupported (benchmark results can
only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. 
Throughput is one thing. Constistent low latency a completely different. The latter is
much much harder.

But what is the "interface" between test cases from "twisted-benchmarks" to codespeed?

This

https://github.com/tobami/codespeed#saving-data

seems to suggest, performance test results are HTTP/POSTed as JSON to codespeed.

And codespeed is then only responsible for visualization and web hosting, right?

I think we can find something better for that part.

> (And if you care a lot about performance in a particular environment you
> could set it up in that environment and get attention for it :)).

Yes, in particular that very last one is a factor to justify efforts;) Anything like having
a promo logo or similar - that would be an argument to invest time and material.
I will seriously contemplate .. need to align with strategy/available time.

We already host FreeBSD buildslaves for both Twisted and PyPy. That might be another
synergy (hosting the latter on that same boxes).

> You should also have a look at the existing benchmark suite and potentially
> look at maintaining / expanding that as well.

I will try to integrate some of this into our upcoming perf. infrastructure.

/Tobias

> 
> Thoughts?
> 
> -glyph
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python