Suites

Name: poly-bench
Author: poly-bench

Suite declaration, suiteType, runMode, sameDataset, and configuration

Suites group related benchmarks with shared configuration, setup code, and fixtures. When you declare a suite, you must specify three header tokens: suiteType, runMode, and sameDataset. These control how benchmarks are run and how results can be visualized.

Declaration Syntax

1declare suite <name> <suiteType> <runMode> sameDataset: <true|false> { ... }

All three header tokens are required.

All three header tokens (suiteType, runMode, sameDataset) are required.

Required Header Tokens

The table below describes each header token and its allowed values.

Token	Values	Description
suiteType	`performance` \| `memory`	`performance` = timing only; `memory` = alloc tracking (B/op, allocs/op)
runMode	`timeBased` \| `iterationBased`	`timeBased` = calibrate via `targetTime`; `iterationBased` = fixed `iterations`
sameDataset	`true` \| `false`	Whether all benchmarks in the suite share the same fixture dataset

Enabled Languages

Only languages that have been set up or initialized in the project's polybench.toml can be used in your benchmarks. If you write setup rust or bench { rust: ... } in a .bench file but Rust is not enabled in the project, you will get a syntax or validation error.

To add or register a new language for use in your bench files, run:

$poly-bench add-runtime rust
$poly-bench add-runtime zig

This updates polybench.toml and creates the .polybench/runtime-env/<lang>/ directory for that runtime. You can then add setup blocks and benchmark implementations for that language.

Run poly-bench add-runtime <lang> to register a language before using it in setup or bench blocks.

suiteType

The suiteType determines whether you are measuring timing only or also tracking allocations.

Value	Effect
`performance`	Standard timing benchmarks. Output: ns/op, ops/s.
`memory`	Enables allocation tracking. Output includes B/op, allocs/op. Requires `alloc_tracker` for Rust.

1declare suite hashBench performance timeBased sameDataset: true {
2  baseline: "go"
3  warmup: 100
4  targetTime: 1000ms
5
6  setup go { ... }
7  setup ts { ... }
8  bench keccak256 { go: hash(data)  ts: hash(data) }
9}

runMode

The runMode controls how many iterations each benchmark runs. You can either calibrate to a target time or use a fixed iteration count.

Value	Use	Key property
`timeBased`	Calibrate iterations to hit a target wall-clock time	`targetTime` (e.g. `1000ms`)
`iterationBased`	Fixed iteration count per benchmark	`iterations` (e.g. `100000`)

Use timeBased for most suites. It adapts to benchmark speed and avoids per-benchmark drift.

1declare suite mySuite performance timeBased sameDataset: true {
2    targetTime: 3000ms   # Calibrate to ~3s per benchmark
3    warmup: 100
4
5  # ... setup, fixtures, benchmarks ...
6}

With timeBased, do not set iterations. With iterationBased, do not set targetTime.

sameDataset

The sameDataset flag indicates whether all benchmarks in the suite operate on the same fixture set.

Value	Use
`true`	All benchmarks operate on the same fixture set. Required for line/bar charts.
`false`	Benchmarks may use different fixtures.

Use sameDataset: true when you want to use bar charts and line charts. These chart types are only available when all benchmarks in the suite are derived from the same dataset — they plot performance across benchmarks that share a common x-axis (e.g. input size). Benchmarks that do not use the same data, or that are merely grouped by a similar topic (e.g. different hash functions or unrelated operations), are best visualized with speedup charts, which compare languages relative to a baseline without requiring a shared dataset.

Use sameDataset: true when you want bar/line charts; they require a shared dataset across benchmarks.

1declare suite sortBench performance timeBased sameDataset: true {
2  # All benchmarks (n100, n200, n500) share fixtures s100, s200, s500
3  fixture s100 { hex: @file("fixtures/sort_100.hex") }
4  fixture s200 { hex: @file("fixtures/sort_200.hex") }
5  fixture s500 { hex: @file("fixtures/sort_500.hex") }
6
7  bench n100 { go: sort(s100)  ts: sort(s100) }
8  bench n200 { go: sort(s200)  ts: sort(s200) }
9  bench n500 { go: sort(s500)  ts: sort(s500) }
10
11  after { charting.drawLineChart(...) }  # Requires sameDataset: true
12}

Suite Configuration Properties

Beyond the header tokens, suites support a number of configuration properties. The table below lists them all.

Property	Type	Default	Description
`description`	string	—	Human-readable description
`warmup`	number	`1000`	Warmup iterations before timing
`targetTime`	duration	`3000ms`	Target time for `timeBased` mode
`iterations`	number	—	Fixed iterations for `iterationBased` mode
`count`	number	`1`	Number of timed runs per benchmark
`cvThreshold`	number	`5.0`	Target coefficient of variation (%)
`outlierDetection`	boolean	`true`	IQR-based outlier removal
`baseline`	string	—	Language for speedup ratios (`"go"`, `"ts"`, `"rust"`, etc.)
`order`	string	`sequential`	`sequential`, `parallel`, or `random`
`timeout`	duration	—	Suite-level timeout
`requires`	string[]	`[]`	Languages every benchmark must implement
`fairness`	string	`"strict"`	`"strict"` or `"legacy"`
`fairnessSeed`	number	—	Deterministic seed for fairness randomization
`sink`	boolean	`true`	Black-box sink to prevent dead-code elimination

Duration Syntax

Unit	Example
`ms`	`3000ms`
`s`	`5s`
`m`	`1m`

Suite-level `after` Block

The suite-level after block runs after all benchmarks complete. It is the correct place for chart generation. Requires use std::charting.

1use std::charting
2
3declare suite mySuite performance timeBased sameDataset: true {
4  baseline: "go"
5  # ... setup, fixtures, benchmarks ...
6
7  after {
8      charting.drawTable(
9          title: "Performance Comparison",
10          output: "results.svg"
11      )
12      charting.drawSpeedupChart(
13          title: "Speedup vs Go",
14          output: "speedup.svg"
15      )
16  }
17}

Line and bar charts require sameDataset: true and at least 2 benchmarks.

Complete Example

1use std::charting
2
3declare suite bubbleN performance timeBased sameDataset: true {
4  description: "O(n²) bubble sort on int32 array"
5  baseline: "go"
6  warmup: 100
7  targetTime: 1000ms
8  fairness: "strict"
9  cvThreshold: 5
10  count: 1
11
12  setup go { ... }
13  setup ts { ... }
14  setup rust { ... }
15
16  fixture s100 { hex: @file("fixtures/sort/sort_100.hex") }
17  fixture s200 { hex: @file("fixtures/sort/sort_200.hex") }
18
19  bench n100 { go: bubble(s100)  ts: bubble(s100)  rust: bubble(&s100) }
20  bench n200 { go: bubble(s200)  ts: bubble(s200)  rust: bubble(&s200) }
21
22  after {
23      charting.drawLineChart(
24          title: "Bubble Sort",
25          output: "bubble-line.svg",
26          yScale: "linear"
27      )
28  }
29}