# LlamaStash benchmarks

LlamaStash spawns the **unmodified upstream `llama-server`**. Three distinct questions follow from that, and there's a suite for each:

| Suite | Question | Lives in |
|---|---|---|
| **A — overhead regression** | Does `llamastash start <model>` add measurable overhead on top of raw `llama-server` for the same effective argv? | [`docs/benchmarks/overhead/`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/overhead/) |
| **B — cross-tool comparison** | How does LlamaStash-as-shipped compare to Ollama and LM Studio on the same model, same hardware, driven through their OpenAI-compatible HTTP endpoints? | [`docs/benchmarks/runs/`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/runs/) |
| **C — proxy overhead** | Does going through the LlamaStash OpenAI-compat proxy cost anything vs hitting the same `llama-server` directly? | [`docs/benchmarks/proxy/`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/proxy/) |

Both suites are driven by the harness under `scripts/bench/`. Per-cell JSONs are checked into the repo so every published chart is reproducible from source — see [§Re-running](#re-running) below.

Methodology, fairness notes, the variance gate, and the cross-backend determinism caveat live in [`docs/benchmarks/methodology.md`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/methodology.md). Read it before quoting any single number — without context, individual cells mislead.

## Cross-tool benchmarks

Each cross-tool run pits LlamaStash against raw `llama-server`, Ollama, and LM Studio on the same hardware, same model bytes, same workloads. We publish a curated report per run on the chronological [results index](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/index.md); the most recent run's headline lives here.

### Apple M1 - macOS

**Hardware:** Apple M1 · 16 GB unified memory · macOS 26 (Darwin 25.4.0) · Metal backend  
**Date:** 2026-05-27 · **llama.cpp build:** `9330 (328874d05)` (Homebrew, Metal build)  
**Tools:** LlamaStash 0.0.1, raw `llama-server` (same Homebrew binary), Ollama 0.24.0, LM Studio (Metal runtime)  
**Workloads:** `chat_turn`, `agent_decode`, `rag_prefill`, `parallel_4` (1 warmup + 4 measured reps per cell)

| Tool | small (Qwen2.5-0.5B Q4) |
|---|---:|
| **LlamaStash** | **95.6 / 18** ✦ |
| raw `llama-server` | 91.9 / 20 |
| LM Studio | 88.4 / 68 |
| Ollama 0.24.0 | 79.6 / 102 |

Each cell is **decode tok/s / TTFT ms**, averaged across `defaults` + `normalized` modes on the `chat_turn` workload.

![Apple M1 decode tok/s by tool, defaults vs normalized](../assets/charts/03-m1-decode-defaults-vs-normalized.png)

✦ In `defaults` mode LlamaStash reaches 99.0 tok/s / 15 ms TTFT vs raw `llama-server`'s 92.3 / 19 — a 7% decode lead from the Metal defaults overlay in `defaults_table.rs`. Normalized mode collapses the gap to <1% (92.2 vs 91.5), confirming zero wrapper overhead.

**Highlights:**

- **LlamaStash ≡ raw `llama-server`** in normalized mode (within ≤1%), and **beats it in defaults mode** by 7% thanks to Metal-aware hardware defaults.
- **Ollama RAG TTFT: 2.8 s** per request vs 40–55 ms for the direct llama-server path — ~52× slower. No prefix caching in its default configuration.
- **Proxy overhead: zero** — Suite C measured −0.6 ms TTFT and +1.4% decode (within noise) going through the proxy vs hitting `llama-server` directly.

Full per-workload tables and six findings → [`docs/benchmarks/macos-m1-final-report.md`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/macos-m1-final-report.md).

### AMD APU - Linux

**Hardware:** AMD Ryzen AI Max+ 395 ("Strix Halo") · Radeon 8060S iGPU (RDNA 3.5, `gfx1151`) · 121 GiB unified RAM · 70 W TDP · Linux  
**Dates:** 2026-05-23 (small), 2026-05-24 (mid, large_dense, large_moe), 2026-06-01 (Vulkan addendum)  
**Toolchain:** system ROCm 7.2.3 + self-built `llama-server` `b9440 (e6123e208)` (HIP and Vulkan)  
**Tools:** LlamaStash, raw `llama-server` (same binary), Ollama 0.24.0, LM Studio 2.18.0  
**Workloads:** `chat_turn`, `agent_decode`, `rag_prefill`, `parallel_4` (1 warmup + 3-4 measured reps per cell, variance-gated at 10%)

#### HIP/ROCm main table — `chat_turn` `normalized` mode

| Tool               | small (E2B Q4) |  mid (31B Q4) | large_dense (27B Q8) | large_moe (35B-A3B Q8) | Engine notes |
|---|---:|---:|---:|---:|---|
| **LlamaStash**     |  **82.1 / 51** | **9.9 / 468** |        **7.5 / 406** |         **42.3 / 178** | local HIP/ROCm |
| raw `llama-server` |      81.0 / 51 |     9.9 / 466 |            7.5 / 406 |             43.1 / 185 | local HIP/ROCm |
| LM Studio 2.18.0   |     91.1 / 187 |    — (crash¹) |           — (crash¹) |             — (crash¹) | bundled ROCm 6.4 vendor |
| Ollama 0.24.0      |     50.8 / 224 |    4.8 / 1096 |           2.6 / 1750 |             12.2 / 484 | bundled |

Decode tok/s / TTFT ms on `chat_turn`, `normalized` mode (`ctx=4096`, `n_gpu_layers=999`, `flash_attn=on`, `batch=512`, `ubatch=512`). Each row from one bench JSON (no averaging across runs or modes).

![AMD APU decode by tool (HIP/ROCm, chat_turn defaults)](../assets/charts/01-amd-apu-decode.png)

![AMD APU TTFT by tool (HIP/ROCm, chat_turn defaults, log scale)](../assets/charts/02-amd-apu-ttft.png)

¹ LM Studio's bundled ROCm vendor libraries (v6.4 in `linux-llama-rocm-vendor-v3`) abort in `ggml_cuda_error` during backend initialization on `gfx1151` (Strix Halo), across all LMS-shipped runtime versions (`amd-rocm-avx2@1.33.0`, `@2.16.0`, `@2.18.0`). The system ROCm 7.2.3 loads the same models without issue via raw `llama-server`, so this is an LM Studio vendor-bundle limitation, not a hardware limitation. See [Finding #2 of the full report](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/linux-amd-apu-final-report.md#2-engine-choice-hip-vs-vulkan-is-workload--and-model-size-dependent--not-a-simple-vulkan-wins).

#### Vulkan addendum — LlamaStash vs LM Studio

| Tool             | small (E2B Q4) |   mid (31B Q4) | large_dense (27B Q8) | large_moe (35B-A3B Q8) |
|---|---:|---:|---:|---:|
| **LlamaStash**   | **101.2 / 55** | **10.8 / 671** |            7.5 / 196 |          **50.7 / 72** |
| LM Studio 2.18.0 |     93.6 / 191 |     7.1 / 2307 |        **8.0 / 801** |             38.4 / 227 |

Vulkan via self-built `llama-server b9440 Vulkan` (LlamaStash) and bundled `vulkan-avx2@2.18.0` (LM Studio). Same GGUF bytes. raw `llama-server` and Ollama omitted (wrapper-overhead claim already covered above; mainline Ollama has no Vulkan support).

![AMD APU Vulkan addendum: LlamaStash vs LM Studio (chat_turn defaults)](../assets/charts/05-amd-apu-vulkan-addendum.png)

#### Combined view — all tools × backends × modes

![AMD APU combined: chat_turn across HIP/ROCm + Vulkan, defaults + normalized](../assets/charts/06-amd-apu-combined.png)

![AMD APU combined: agent_decode across HIP/ROCm + Vulkan, defaults + normalized](../assets/charts/07-amd-apu-combined-agent-decode.png)

Light mauve / dark mauve = HIP/ROCm defaults / normalized. Light teal / dark teal = Vulkan defaults / normalized. Em-dashes mark cells not run (raw and Ollama on Vulkan) or backend-init crashes (LM Studio on HIP for `mid`, `large_dense`, `large_moe`). `agent_decode` is the longer-context workload (256-token target vs 64 for `chat_turn`).

**Highlights:**

- **LlamaStash ≡ raw `llama-server`** within ≤1% on 3 of 4 cells (`small` +1.3%, `mid` −0.0%, `large_dense` +0.1%). `large_moe` shows −1.8%, the one cell outside the ±1% advisory band but well inside the ±2% catastrophic floor; cell flips to +1.1% LlamaStash in `defaults` mode, reading as wrapper-attribution noise on a MoE workload.
- **Ollama** is 38–72% slower decode than raw `llama-server`, and **RAG is catastrophic** (cold prefill every rep — 17 s on small, ~4 min on mid 31B).
- **LM Studio HIP/ROCm only loads `small`** on `gfx1151`; full per-size comparison requires the Vulkan addendum. LMS Vulkan `mid` regressed from `11.6 tok/s` in May to `7.1 tok/s` after the LMS app + bundled runtime upgrade (`vulkan-avx2@2.16.0` → `@2.18.0`); reproducible across two memory-state runs.

Full per-workload tables, engine A/B, and seven findings → [`docs/benchmarks/linux-amd-apu-final-report.md`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/linux-amd-apu-final-report.md).  
Raw per-cell data → [`docs/benchmarks/runs/`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/runs/).  
Regenerate this table any time: `make bench-table`.

### NVIDIA RTX 3050 Ti - Linux

**Hardware:** NVIDIA GeForce RTX 3050 Ti Laptop GPU (Ampere sm\_86, **4 GiB VRAM**) · Intel i9-11900H · 63 GiB RAM · Manjaro Linux · NVIDIA driver 595.71.05 / CUDA 13.2  
**Date:** 2026-05-28 · **llama.cpp build:** `b9360 (6b4e4bd5)` (CUDA, self-built) + `b9360` prebuilt Vulkan asset for the Vulkan lane  
**Tools:** LlamaStash 0.0.1, raw `llama-server` (same binaries), Ollama 0.24.0, LM Studio 2.16.0  
**Workloads:** `chat_turn`, `agent_decode`, `rag_prefill`, `parallel_4` (1 warmup + 4 measured reps per cell)

`defaults` mode headline (out-of-box experience); `normalized` mode shows a different picture for some tools — see full report.

| Tool | CUDA | Vulkan |
|---|---:|---:|
| **LlamaStash** | **41.1 / 74** | **42.0 / 113** |
| raw `llama-server` | 36.6 / 110 | 37.5 / 148 |
| LM Studio 2.16.0 | **48.7 / 318** | **48.3 / 308** |
| Ollama 0.24.0 | 40.7 / 120 | 42.0 / 115 |

Each cell is **decode tok/s / TTFT ms** on `chat_turn`, `defaults` mode. (CUDA/Vulkan here refer to the engine used by each tool, explicitly pinned per the methodology.)

![NVIDIA RTX 3050 Ti decode tok/s by tool, CUDA vs Vulkan, defaults](../assets/charts/04-nvidia-decode-cuda-vs-vulkan.png)

**Highlights:**

- **Vulkan decode ≥ CUDA decode** in 26 of 28 comparable cells (median +5%) — the "CUDA wins on NVIDIA" intuition does not hold for bandwidth-bound Q3 decode on a 4 GiB Ampere card.
- **LlamaStash leads raw `llama-server` in defaults mode** (12–16% decode, 33–49% TTFT) because its defaults overlay injects hardware-aware knobs. Normalized mode: within ≤0.5 tok/s.
- **Ollama Vulkan rag\_prefill timed out** — wall-clock exceeded 1.5 h for two cells. CUDA rag\_prefill is slow but workable (3.4–3.7 s TTFT).

Full per-workload tables, CUDA vs Vulkan engine A/B, and six findings → [`docs/benchmarks/linux-nvidia-final.md`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/linux-nvidia-final.md).

## Overhead regression (Suite A)

Suite A runs `llamastash start <model>` and raw `llama-server` back-to-back with the same effective argv, then compares deltas against a two-tier threshold:

| Metric | Catastrophic (exits non-zero) | Advisory (exits zero with banner) |
|---|---|---|
| `ttft_ms` delta | ≥ 200 ms | ≥ 30 ms |
| `decode_tps` delta percentage | ≥ 2.0% slower | ≥ 0.5% slower |
| Daemon idle RSS | ≥ 64 MiB extra | ≥ 48 MiB extra |

The orchestrator also asserts argv **byte-equality** (after stripping `--port`) so there's no place for a hidden tweak to hide. Per-host results land in [`docs/benchmarks/overhead/<host-id>/`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/overhead/). Thresholds are tunable in `scripts/bench/overhead/thresholds.json`.

![Suite A overhead: LlamaStash vs raw llama-server across six matched-flags cells, with advisory and hard-fail lines](../assets/charts/08-suite-a-overhead.png)

Top: decode delta % per cell, positive = LlamaStash faster. Bottom: TTFT delta ms. Yellow dashed lines are the advisory thresholds (±0.5% decode, ±30 ms TTFT); red dashed lines are the hard-fail thresholds (±2% decode; ±200 ms TTFT is off-chart). All six cells inside the hard-fail box.

## Proxy overhead (Suite C)

Suite C answers a simple question: if you talk to the LlamaStash proxy instead of `llama-server` directly, does it cost you anything?

The harness brings up one model, then sends the same chat request to both URLs (direct port + proxy on `127.0.0.1:11434`), alternating one-for-one. Same `llama-server` behind both.

Results so far:

- **AMD APU** (`deepu-flowz13-arch`, gemma-4-E2B-it-Q4\_K\_M, 15 requests each side): TTFT +0.45 ms at the median (52.37 → 52.82 ms), tokens/sec unchanged (92.80 → 92.70).
- **Apple M1** (`macos-m1`, Qwen2.5-0.5B-Instruct Q4\_K\_M, 5 measured reps per phase): TTFT −0.6 ms (within noise), decode +1.4% (within noise).
- **NVIDIA RTX 3050 Ti** (`deepu-xps`, gemma-3-4b-it.Q3\_K\_M, Vulkan lane, 4 measured reps per phase): TTFT +0.57 ms, decode −0.20% (within noise).

![Suite C proxy overhead: decode delta % (top) and TTFT delta ms (bottom) across AMD APU, Apple M1, RTX 3050 Ti](../assets/charts/09-suite-c-proxy.png)

TTFT panel is zoomed to ±1.6 ms so the sub-millisecond deltas are visible (advisory ±30 ms and hard-fail ±200 ms both off-chart at that zoom).

On all three platforms: **zero measurable proxy cost.** Full numbers and method → [`docs/benchmarks/proxy/results.md`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/proxy/results.md).

## Re-running

Both suites are maintainer-run; nothing in CI fires them.

```sh
make bench-end-to-end -- --dry-run   # print the planned Suite B matrix
make bench-end-to-end                 # run Suite B (cross-tool)
make bench-overhead                   # run Suite A (overhead)
make bench-proxy -- --model <gguf>    # run Suite C (proxy vs direct)
make bench-test                       # harness unit tests only — no benchmark spawn
make bench-table                      # pivot existing JSONs into the headline summary
```

Prerequisites and per-backend gotchas: [`methodology.md §Re-running`](https://github.com/llamastash/llamastash/blob/main/docs/benchmarks/methodology.md#re-running).  
Honored env vars (host-id override, model overrides, port base, readiness timeout, Ollama keep-imports): same section.

## Raw data layout

```
docs/benchmarks/
├── runs/                         # Suite B per-host JSONs (cross-tool)
│   └── <host-id>/<YYYY-MM-DD>-<hms>-<sha>.json
├── overhead/                     # Suite A per-host JSONs (overhead regression)
│   └── <host-id>/<YYYY-MM-DD>-<sha>.json
├── proxy/                        # Suite C per-host JSONs (proxy vs direct)
│   └── <host-id>/<YYYY-MM-DD>-<sha>.json
├── methodology.md                # The fairness contract — read first
├── index.md                      # Chronological index of all published runs
├── macos-m1-final-report.md         # Curated Apple M1 run
├── linux-nvidia-final.md      # Curated NVIDIA RTX 3050 Ti run (CUDA + Vulkan)
├── linux-amd-apu-final-report.md   # Curated AMD APU run
└── results-<YYYY-MM-DD>.md         # Auto-rendered raw-data pages per run day
```

Each JSON validates against the v1 schema in [`scripts/bench/end_to_end/schema.py`](https://github.com/llamastash/llamastash/blob/main/scripts/bench/end_to_end/schema.py). The auto-rendered dated pages are reproducible from the JSONs via `scripts/bench/end_to_end/render.py`; the `bench-table` headline is reproducible via `scripts/bench/end_to_end/table.py`. Community contributors can drop a new host directory under `runs/` and re-render — no central database, no schema migration dance.
