# Benchmarking and Profiling Guide

## Quick Start

### Running Benchmarks

```bash
# Download test documents (first time only)
cd benches/documents && ./download.sh && cd ../..

# Run benchmarks
cargo bench --bench formatting

# Run LSP incremental didChange benchmarks
cargo bench --bench lsp_incremental

# Run interned key impact benchmark
cargo bench --bench interned_keys

# Run CLI cache cold-vs-warm benchmark
cargo bench --bench cli_cache

# Generate docs + machine-readable JSON
./benches/generate_docs.sh
```

### Profiling

For line-level profiling with flame graphs:

```bash
# Install flamegraph
cargo install flamegraph

# Profile the benchmark
cargo flamegraph --bench formatting

# Opens flamegraph.svg showing hotspots
```

For large-document bottlenecks (e.g. Pandoc MANUAL):

```bash
# Ensure pandoc_manual.md exists first
cd benches/documents && ./download.sh && cd ../..

# Profile only the selected document
PANACHE_BENCH_DOC=pandoc_manual.md PANACHE_BENCH_ITERATIONS=3 \
    cargo flamegraph --bench formatting

# LSP incremental benchmark knobs
# PANACHE_LSP_BENCH_CAP=8 (default)
# PANACHE_LSP_BENCH_ITERATIONS=80 (default)
# PANACHE_LSP_BENCH_OUTPUT_JSON=benches/lsp_incremental_results.json
```

For more detailed profiling:

```bash
# Linux perf (CPU profiling)
perf record --call-graph dwarf cargo bench --bench formatting
perf report

# Line-level annotation for selected stress document
PANACHE_BENCH_DOC=pandoc_manual.md PANACHE_BENCH_ITERATIONS=3 \
    perf record --call-graph dwarf cargo bench --bench formatting
perf annotate

# Valgrind (memory profiling)
valgrind --tool=cachegrind cargo bench --bench formatting
```

## Benchmark Infrastructure

### Document Management

- **`benches/documents/`**: Test documents for benchmarking
  - `small.qmd`: Committed baseline (747 bytes)

  - `pandoc_manual.md`: Stress-test doc downloaded from upstream pandoc
    `MANUAL.txt`

  - Other files: Downloaded on-demand from Quarto docs

  - `.gitignore`: Excludes downloaded files from repo
- **`benches/documents/download.sh`**: Downloads real Quarto documents
  - Reproducible: same sources every time
  - Lightweight: doesn't bloat repo

### Benchmark Code

- **`benches/formatting.rs`**: Main benchmark suite
  - Tests parse, format, and full pipeline

  - Multiple document sizes and types

  - Reports throughput in KB/s
- **`benches/interned_keys.rs`**: Key interning measurement harness
  - Compares owned vs interned key map build costs
  - Reports repeated-byte potential from duplicated keys
- **`benches/cli_cache.rs`**: CLI cache warm-hit benchmark harness
  - Compares uncached (cold) vs cached (warm) runs for `format --check` and
    `lint`
  - Tunable with `PANACHE_CLI_CACHE_BENCH_FILES` and
    `PANACHE_CLI_CACHE_BENCH_ITERATIONS`
  - Optional JSON output via `PANACHE_CLI_CACHE_BENCH_OUTPUT_JSON`
- **`benches/compare_all.sh`**: Multi-formatter comparison
  - Compares panache vs Prettier vs Pandoc vs rumdl across six documents (small,
    medium, tables, math, large, and the full Pandoc manual)
  - **Text mode (default)**: prints colored results and appends to
    `benchmark_results.txt`
  - **JSON mode**: `bash benches/compare_all.sh --json [--out PATH]` writes
    structured JSON consumed by `docs/guide/performance.qmd`. Default output
    path is `docs/guide/performance_data.json`.
  - Prefers [hyperfine](https://github.com/sharkdp/hyperfine) for stats when
    available (and `jq` for parsing); otherwise falls back to a simple shell
    timing loop emitting mean only (`stddev_ms`/`min_ms`/`max_ms` are `null`).
  - Drives the data-driven `docs/guide/performance.qmd` page; `freeze: true` on
    that doc means the benchmark only re-runs when you delete
    `docs/_freeze/guide/performance/` and re-render.
- **`benches/generate_docs.sh`**: Captures results for documentation
  - Generates `benches/benchmark_results.json` (machine-readable)
  - Renders `docs/benchmarks.qmd` from JSON
  - Deterministic output for CI checks

## What to Benchmark

Good targets for benchmarking: - **Full pipeline** (parse + format) - what users
experience - **Parse speed** - CST construction - **Format speed** - CST
traversal and output - **Document types** - simple text vs complex (tables,
math, divs) - **Document sizes** - small (1KB), medium (10-50KB), large (100KB+)

## Performance Tips

Current performance baseline: - \~20MB/s throughput on typical documents - \~1ms
to format a 30KB document - Parse takes \~30-40% of time, format \~60-70%

To improve performance, profile with flamegraph to find hotspots.

## Adding New Benchmarks

1. Add document to `benches/documents/` (or update `download.sh`)
2. Load in `benches/formatting.rs` with `load_document()`
3. Call `run_benchmark()` with appropriate iteration count
4. Run and verify results

## Integrating with Docs

After running benchmarks:

```bash
# Generate fresh benchmark page
./benches/generate_docs.sh

# Verify tracked artifacts are up to date (CI-friendly)
./benches/check_docs.sh

# Preview in Quarto
cd docs && quarto preview

# Commit to repo
git add benches/benchmark_results.json docs/benchmarks.qmd
git commit -m "docs: update benchmark results"
```
