# AR-benchmarks: instruction-counting benchmarks for the hot CLI commands

How `grund` turns the [§GOAL-fast-feedback](../goals.md#goal-fast-feedback-grund-must-be-as-fast-as-possible) budget into a number a machine records on every commit. [§GOAL-fast-feedback.1](../goals.md#1-performance-targets) writes the targets down (under 100 ms on this repo; under 1 s on a 10k-file repo; at most one allocation per file) and [§GOAL-fast-feedback.3](../goals.md#3-measurable) asks CI to "track the number across commits and fail on regression" — but until something measures, the budget is a promise. This spec pins what is measured, with what tool, and why the meter counts CPU instructions rather than wall-clock seconds. The roadmap item that owns the remaining work — a synthetic large fixture, committed baselines, and the CI gate that fails the build on a regression — is [§RM-benchmarks](../roadmap.md#rm-benchmarks-a-benchmark-harness-for-the-goal-fast-feedback-budgets); the decision that picks instruction counting over a wall-clock harness is [§DA-benchmark-instruction-counting](../decisions/architectural/DA-benchmark-instruction-counting.md#da-benchmark-instruction-counting-the-performance-harness-counts-instructions-not-wall-clock-seconds).

## 1. What is benched

A `cargo bench` harness at `benches/instructions.rs`, gated behind the `bench` Cargo feature so a plain `cargo test --all-targets` compiles only a no-op bench target and never tries to run the Callgrind body (it needs Valgrind). Each benchmark runs the **freshly built `grund` binary** as a subprocess under Callgrind — not a library call — because the figure we care about is the cost of an invocation an agent or a CI step actually makes, including process start-up, argument parsing, the walk, and output formatting.

Every benchmark scans this repository's own conformant tree (`CARGO_MANIFEST_DIR`), the same input the `grund .` self-host loop in CI already exercises ([§RM-self-host](../roadmap.md#rm-self-host-guard-the-self-host-loop-in-ci)). The benched subcommands are the ones the agent loop and the CI/pre-commit gate run most:

| Benchmark | Invocation | Who runs it, how often |
|---|---|---|
| `check` | `grund check <repo>` | Every save (watch / LSP), every commit (pre-commit), every push (CI). The headline operation [§GOAL-fast-feedback](../goals.md#goal-fast-feedback-grund-must-be-as-fast-as-possible) exists for. |
| `list` | `grund list <repo>` | An agent discovering the right `<ID>` before it cites one. |
| `show --brief` | `grund <ID> --brief <repo>` | An agent or LSP hover checking the title plus first paragraph. |
| `show` | `grund <ID> <repo>` | An agent grounding itself on a bare citation — the lead-default read AGENTS.md / CLAUDE.md tell it to do first, before editing the code that realizes a cited ID. |
| `show --full` | `grund <ID> --full <repo>` | An agent escalating to the full declaration body when the narrower slices are not enough. |
| `refs` | `grund refs <ID> <repo>` | An agent checking a declaration's blast radius before changing or moving it. |
| `cover` | `grund cover <repo>` | What the diff-aware co-change recipe ([§RM-cochange-gate](../roadmap.md#rm-cochange-gate-a-pre-commit--ci-recipe--no-impl-change-without-spec-and-test)) and CI consume. |
| `fmt --check` | `grund fmt --check <repo>` | The pre-commit / CI normalization gate. |

`check`, `list`, `cover`, and `fmt --check` walk the whole scanned tree; `show --brief`, `show`, `show --full`, and `refs` do the same walk plus a per-ID extraction or filter — so the set spans the engine's two cost shapes (full report vs. single-ID query), and the show ladder spans the short hover preview, the cheap lead default, and the full recursive body over realistic inputs. Each command exits 0 on this conformant tree (which keeps iai-callgrind's default `ExitWith::Success` happy); on a broken tree `check` / `fmt --check` exit non-zero and the bench fails — intentionally, since a baseline recorded against a broken tree is worthless.

This same workload is the training run for the profile-guided-optimization build of the distributed release binary (`scripts/pgo-build.sh`, [§DA-pgo-release](../decisions/architectural/DA-pgo-release.md#da-pgo-release-distributed-binaries-are-pgo-built-trained-on-the-benchmark-workload)): the thing we benchmark is the thing we profile against, so "the operations that matter" has one definition. PGO stays out of the normal development build and push/PR CI loop; it is run by the release pipeline, and by explicit benchmark work when comparing the optimized release artifact. The command list is duplicated in two forms — the `#[binary_benchmark]` set in `benches/instructions.rs` and the shell loop in `scripts/pgo-build.sh` — each with a comment pointing at the other.

## 2. Why instruction counts, not wall-clock

Callgrind reports the number of CPU instructions executed (plus cache-miss estimates), and that number is **deterministic** for a given binary and input — it does not move when the CI runner is loaded, when a neighbour VM is noisy, or when the kernel reschedules. A wall-clock harness (criterion and friends) measures the thing the [§GOAL-fast-feedback.1](../goals.md#1-performance-targets) budget is literally written in (milliseconds), but its variance on shared CI is large enough that "fail on regression" either flakes (tight threshold) or misses real regressions (loose threshold). Instruction count is the proxy that makes [§GOAL-fast-feedback.3](../goals.md#3-measurable)'s "fail on regression" implementable: a quadratic walk or an accidental second read pass moves the instruction count by a margin no scheduling noise can mask. The full rationale, and the alternatives weighed, is [§DA-benchmark-instruction-counting](../decisions/architectural/DA-benchmark-instruction-counting.md#da-benchmark-instruction-counting-the-performance-harness-counts-instructions-not-wall-clock-seconds).

The millisecond budget itself is still backstopped by the cheap wall-clock guard in [§AR-ci.4](AR-ci.md#4-performance-smoke-guard) (the built `grund .` self-check under a generous `timeout`); the instruction-count harness is the precise regression meter, the timeout is the catastrophic-blowup floor. The two are complementary, not redundant.

## 3. How it runs

- Locally: `cargo bench --features bench` (requires `valgrind` and `iai-callgrind-runner` on `PATH`). With no prior run, iai-callgrind records the counts and exits clean; on a later run in the same `target/` it prints the delta against the previous run.
- In CI: a dedicated `bench` job installs `valgrind` and `cargo install iai-callgrind-runner` (version-pinned to the `iai-callgrind` dev-dependency), then runs `cargo bench --features bench` — see [§AR-ci.5](AR-ci.md#5-benchmark-job). Today the job runs the harness and surfaces the counts; turning a committed-baseline regression threshold into a build-failing check is the remaining [§RM-benchmarks](../roadmap.md#rm-benchmarks-a-benchmark-harness-for-the-goal-fast-feedback-budgets) work, at which point this job becomes the authoritative [§GOAL-fast-feedback](../goals.md#goal-fast-feedback-grund-must-be-as-fast-as-possible) meter and the [§AR-ci.4](AR-ci.md#4-performance-smoke-guard) timeout stays as a backstop.

## 4. Relationship to the rest of the tree

- [§GOAL-fast-feedback.3](../goals.md#3-measurable) — this harness is the "track the number across commits" mechanism that goal's "Measurable" section calls for; [§GOAL-fast-feedback.1](../goals.md#1-performance-targets)'s budgets are what a committed baseline pins it against.
- [§RM-benchmarks](../roadmap.md#rm-benchmarks-a-benchmark-harness-for-the-goal-fast-feedback-budgets) — owns the not-yet-done parts: the generated 10k-file synthetic fixture (shared with the [§GOAL-small-and-large.5](../goals.md#5-measurable) large-repo fixture and the [§RM-self-host](../roadmap.md#rm-self-host-guard-the-self-host-loop-in-ci) nested-fixtures case), the committed 0.1.0 baseline figures, and the CI regression gate.
- [§DA-benchmark-instruction-counting](../decisions/architectural/DA-benchmark-instruction-counting.md#da-benchmark-instruction-counting-the-performance-harness-counts-instructions-not-wall-clock-seconds) — the architectural decision record for choosing Callgrind/iai-callgrind over a wall-clock criterion harness, and for benchmarking the binary rather than the library.
- [§AR-ci.4](AR-ci.md#4-performance-smoke-guard) / [§AR-ci.5](AR-ci.md#5-benchmark-job) — the interim wall-clock smoke guard and the benchmark CI job, respectively.
- [§RM-core-cli-split](../roadmap.md#rm-core-cli-split-split-grund-core-from-grund-cli) / [§RM-distribution](../roadmap.md#rm-distribution-cargo--npm--pypi-from-one-engine) — the harness is deliberately landed against today's single-binary 0.1.0 build, so when the workspace split and the napi-rs / PyO3 bindings move the engine around, any slowdown shows up as a diff against a known-good instruction count rather than going unnoticed.
- [§DA-pgo-release](../decisions/architectural/DA-pgo-release.md#da-pgo-release-distributed-binaries-are-pgo-built-trained-on-the-benchmark-workload) — reuses this workload as the PGO training corpus for the distributed release binary; the release pipeline that runs `scripts/pgo-build.sh` is owned by [§RM-distribution](../roadmap.md#rm-distribution-cargo--npm--pypi-from-one-engine), the release contract is [§FS-distribution.4](../functional-spec/FS-distribution.md#4-release-process), and normal development CI does not run PGO ([§AR-ci.6](AR-ci.md#6-pgo-stays-out-of-development-ci)).

## 5. Scope

This is a build-and-CI tool, not a CLI surface: it adds no subcommand, no flag, no output shape, so the `e2e/cases/*` corpus is unchanged and nothing here is covered by the [§GOAL-no-silent-breakage](../goals.md#goal-no-silent-breakage-changes-ship-through-a-deprecation-path) user-visible-surface contract. The benchmark *file* is not part of the scanned tree (`benches/` is outside `[scan] include`), so it carries no citations and is exempt from the [§FS-check.3.6](../functional-spec/FS-check.md#36-ungrounded-source-file-opt-in) grounding floor; the benched commands are described above by reference to their functional specs ([§FS-check](../functional-spec/FS-check.md#fs-check-grund-validates-every-reference-in-a-repo), [§FS-cli](../functional-spec/FS-cli.md#fs-cli-grunds-command-line-surface-conventions)) rather than re-specified here.
