<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Feb 26, 2013, at 10:05 AM, Peter Westlake <<a href="mailto:peter.westlake@pobox.com">peter.westlake@pobox.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">On Sun, Jan 6, 2013, at 20:22,<span class="Apple-converted-space"> </span><a href="mailto:exarkun@twistedmatrix.com">exarkun@twistedmatrix.com</a><span class="Apple-converted-space"> </span>wrote:<br><blockquote type="cite">On 12:48 am,<span class="Apple-converted-space"> </span><a href="mailto:peter.westlake@pobox.com">peter.westlake@pobox.com</a><span class="Apple-converted-space"> </span>wrote:<br><blockquote type="cite">On Fri, Jan 4, 2013, at 19:58,<span class="Apple-converted-space"> </span><a href="mailto:exarkun@twistedmatrix.com">exarkun@twistedmatrix.com</a><span class="Apple-converted-space"> </span>wrote:<br></blockquote></blockquote>...<br><blockquote type="cite">Codespeed cannot handle more than one result per benchmark.<br><blockquote type="cite"><blockquote type="cite">The `timeit` module is probably not suitable to use to collect the data<br></blockquote></blockquote></blockquote>.....<br><blockquote type="cite"><blockquote type="cite">What method would you prefer?<br></blockquote><br>Something simple and accurate. :) You may need to do some investigation<span class="Apple-converted-space"> </span><br>to determine the best approach.<br></blockquote><br>1. This is simple:<br><br> def do_benchmark(content):<br> t1 = time.time()<br> d = flatten(request, content, lambda _: None)<br> t2 = time.time()<br> assert d.called<br> return t2 - t1<br><br>Do you think it's acceptably accurate? After a few million iterations,<br>the relative error should be pretty small.<br></div></blockquote><div><br></div><div>Well it rather depends on the contents of 'content', doesn't it? :)</div><div><br></div><div>I think we have gotten lost in the weeds here. We talked about using benchlib.py initially, and then you noticed a bug, and it was mentioned that benchlib.py was mostly written for testing asynchronous things and didn't have good support for testing the simple case here, which is synchronous rendering of a simple document. However, one of twisted.web.template's major features - arguably its reason for existing in a world that is practically overrun by HTML templating systems - is that it supports Deferreds. So we'll want that anyway.</div><div><br></div><div>The right thing to do here would be to update benchlib itself with a few simple tools for doing timing of synchronous tasks, and possibly also to just fix the unbounded-recursion bug that you noticed, not to start building a new, parallel set of testing tools which use different infrastructure. That probably means implementing a small subset of timeit.</div><div><br></div><blockquote type="cite"><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">2. For the choice of test data, I had a quick search for benchmarks from<br>other web frameworks. All I found was "hello world" benchmarks, that<br>test the overhead of the framework itself by rendering an empty page.<br>I'll include that, of course.<br></div></blockquote><div><br></div><div>"hello world" benchmarks have problems because start-up overhead tends to dominate. A realistic web page with some slots and renderers sprinkled throughout would be a lot better. Although even better would be a couple of cases - let's say small, large-sync, and large-async - so we can see if optimizations for one case hurt another.</div><div><br></div><div>As Jean-Paul already mentioned in this thread, you can't have more than one result per benchmark, so you'll need to choose a fixed number of configurations and create one benchmark for each.</div><br><blockquote type="cite"><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">3. Regarding option parsing, is there any reason to prefer twisted.python.usage.Options over [...]</div></blockquote><br></div><div>The reason to prefer usage.Options is consistency. That's what we use on Twisted, and there is no compelling reason to use something else. In any case, if there were a compelling reason to use something else, this wouldn't be the place to start; you could start a separate discussion about option parsing. (Warning; a discussion about option parsing would inevitably be a waste of everyone's time and you should under no circumstances do this.)</div><div><br></div><div>All the options that you might need to parse (well, all the options that you _can_ parse, as far as codespeed is concerned) are already implemented by benchlib.py in <a href="http://launchpad.net/twisted-benchmarks">http://launchpad.net/twisted-benchmarks</a>, so there's no point in writing any option-parsing code for this task anyway. The thing to implement would be a different driver() function that makes a few simple synchronous calls without running the reactor.</div><div><br></div><div>-glyph</div></body></html>