Skip to content

Commit 72eedd0

Browse files
Mark speed differences that code layout alone can make as level
Rebuilding the same sources, or adding one function that nothing calls, moved map_int by up to 5%, and the same commit measured 7% apart on alloc_small depending on which runtime's directory built it. The tables marked only differences within twice the run-to-run spread as level, so these read as regressions or wins that the runtime did not make. A speed ratio within 5% of level is now marked "~" too, and BENCHMARKS.md says why.
1 parent 62e9221 commit 72eedd0

2 files changed

Lines changed: 17 additions & 7 deletions

File tree

‎BENCHMARKS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ These numbers are from one Windows machine; Linux and macOS results are still to
8787
- **One benchmark per process.** Each benchmark runs in its own process, so peak memory belongs to that benchmark alone: the peak working set on Windows, the maximum resident set size elsewhere. Each process does 2 untimed warm-up samples and 10 timed samples, and there are 3 processes per benchmark and runtime. The runtimes take turns, so drift on the machine affects both alike.
8888
- **Same work, checked.** Every benchmark returns a checksum of the work it did, and every process under both runtimes has to produce the same one. A change that skips work or computes the wrong answer fails the run instead of looking faster.
8989
- **Spread.** "±" is the median absolute deviation of the sample times, as a percentage of the median. A ratio marked "~" is within twice the larger spread of the two runtimes, so it counts as level.
90+
- **Code layout.** A tight benchmark's speed also moves with where the compiler happens to place code. The same sources built again, or with one function added that nothing calls, measured up to 5% apart on `map_int`, and the same commit measured 7% apart on `alloc_small` depending on which runtime's directory built it. So a speed ratio within 5% of level is marked "~" as well, and a change of that size between two runs of these tables is not a regression on its own.
9091

9192
## The benchmarks
9293

‎bench/Run.hx‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -476,7 +476,7 @@ class Run {
476476
var row = '| ${r.name} | ${r.unit}/s | ' + cells.join(" | ");
477477
if (compare)
478478
row += " | " + ratio(r.byRuntime.get(lib.name).rate, r.byRuntime.get(stock.name).rate,
479-
Math.max(r.byRuntime.get(lib.name).spread, r.byRuntime.get(stock.name).spread));
479+
Math.max(r.byRuntime.get(lib.name).spread, r.byRuntime.get(stock.name).spread), LAYOUT_BAND);
480480
b.add(row + " |\n");
481481
}
482482

@@ -507,7 +507,7 @@ class Run {
507507
row += " | " + [for (rt in runtimes) formatRate(r.byRuntime.get(rt.name).rate, null) + " ±" + percent(r.byRuntime.get(rt.name).spread)].join(" | ");
508508
if (compare)
509509
row += " | " + ratio(r.byRuntime.get(lib.name).rate, r.byRuntime.get(stock.name).rate,
510-
Math.max(r.byRuntime.get(lib.name).spread, r.byRuntime.get(stock.name).spread));
510+
Math.max(r.byRuntime.get(lib.name).spread, r.byRuntime.get(stock.name).spread), LAYOUT_BAND);
511511
}
512512
b.add(row + " |\n");
513513
}
@@ -536,7 +536,7 @@ class Run {
536536
b.add('\n${report.host}; ${env.compiler}, ${env.pointerBits}-bit; Haxe ${report.haxe}; '
537537
+ '${label("libhxcpp")} at ${report.libhxcpp}; ${runs} processes x ${samples} samples per benchmark; ${report.date}.\n');
538538
b.add("Reproduce with `cd bench && haxe --run Run`. Ratios above 1 favour libhxcpp; "
539-
+ "\"~\" marks a difference within the run-to-run spread.\n");
539+
+ "\"~\" marks a difference within the run-to-run spread, or for speed within the 5% that code layout alone can move it.\n");
540540
return b.toString();
541541
}
542542

@@ -572,13 +572,13 @@ class Run {
572572
if (row.measure == "p999")
573573
ratio(s.latency.p999, l.latency.p999, 0);
574574
else
575-
ratio(l.rate, s.rate, Math.max(l.spread, s.spread));
575+
ratio(l.rate, s.rate, Math.max(l.spread, s.spread), LAYOUT_BAND);
576576
}
577577
var bench = row.bench == "" ? "" : ' (`${row.bench}`)';
578578
b.add('| ${row.area} | ${row.what}$bench | $cell |\n');
579579
}
580580
var env = lib.env;
581-
b.add('\nAbove 1 favours libhxcpp; "~" marks a difference within the run-to-run spread. '
581+
b.add('\nAbove 1 favours libhxcpp; "~" marks a difference within the run-to-run spread, or for speed within the 5% that code layout alone can move it. '
582582
+ '${report.host}; ${env.compiler}, ${env.pointerBits}-bit; Haxe ${report.haxe}; ${lib.label} at ${report.libhxcpp}; ${report.date}.\n');
583583
return b.toString();
584584
}
@@ -597,9 +597,18 @@ class Run {
597597
Sys.println('$file updated');
598598
}
599599

600-
static function ratio(better:Float, worse:Float, spread:Float):String {
600+
/**
601+
How far code layout alone moves a tight benchmark's speed: the same
602+
sources built again, or with one function added that nothing calls,
603+
measured 5% apart on map_int, and the same commit 7% apart on
604+
alloc_small depending on which runtime's directory built it. A speed
605+
difference smaller than this says nothing about the runtime.
606+
**/
607+
static inline var LAYOUT_BAND = 0.05;
608+
609+
static function ratio(better:Float, worse:Float, spread:Float, band = 0.0):String {
601610
var r = better / worse;
602-
return (Math.abs(r - 1) <= spread * 2 ? "~" : "") + ratioText(r);
611+
return (Math.abs(r - 1) <= Math.max(spread * 2, band) ? "~" : "") + ratioText(r);
603612
}
604613

605614
static function ratioText(r:Float):String {

0 commit comments

Comments
 (0)