# Svault 0.6.0 — Concurrency / Pressure Simulation

Companion to the [0.6.0 findings register](../findings/0.6.0.md). This records a
deliberate stress run of the Unix daemon to validate behavior under heavy
concurrent load — the thing reviews reason about but rarely exercise. It backs
the maintainer disposition on finding **#8** ("include performance / max-out
tests across many threads") and confirms the concurrency design holds.

## What it does

`daemon_stress_simulation` (in `src/daemon.rs`, `#[ignore]`d so it never runs in
CI or a normal `cargo test`) drives the **real** `serve`/`handle` path — one
thread per connection, the shared `Arc<Mutex>` key store, an AES-256-GCM decrypt
on every `Get` — in two phases:

1. **Sustained concurrent reads** — `THREADS` reader threads each issue `READS`
   `Get` requests against one unlocked vault, opening a fresh socket connection
   per request (the same pattern the CLI/client uses). Every outcome is
   classified:
   - **correct** — returned the right secret value
   - **refused (busy)** — the daemon answered `too many connections`
     (backpressure at the ceiling); acceptable, the real client falls back
   - **conn err (backlog)** — the connect never reached the daemon (OS listener
     backlog drop under connect-churn); a capacity symptom, not a logic bug
   - **wrong (real bug)** — connected, got a response, but it was wrong. **Must
     be zero no matter how hard we push.**
2. **Connection flood** — open many idle connections (connect, never send) to
   push past the ceiling, confirm the daemon refuses cleanly and **recovers**
   once they drain.

The test asserts `wrong == 0` and `recovered == true`; everything else is
reported, not gated.

## How to run

```bash
# Default run (release strongly recommended — debug is ~10x slower):
cargo test --release daemon_stress_simulation -- --ignored --nocapture

# Tunables (env):
SVAULT_STRESS_THREADS=128 \   # parallel reader threads        (default 64)
SVAULT_STRESS_READS=5000 \    # Get requests per thread        (default 2000)
SVAULT_STRESS_FLOOD=256 \     # idle connections in the flood  (default 256)
SVAULT_STRESS_MAXCONN=512 \   # daemon connection ceiling      (default 512)
SVAULT_STRESS_LOG=run.log \   # append the report here too     (default target/stress-report.log)
  cargo test --release daemon_stress_simulation -- --ignored --nocapture
```

A formatted report is printed and appended to `SVAULT_STRESS_LOG`, so a run can
be captured and attached to a review or a performance check.

## Recorded run — 2026-05-29 (svault 0.6.0, release)

Host: Apple M4 Max (the test harness reported 8 available threads in this
sandboxed run). 64 reader threads × 2,000 reads = **128,000 concurrent `Get`s**.

### Run A — shipped default (ceiling 512) + connect retry

```
correct              128000   (100.0%)
refused (busy)       0
conn err (backlog)   0
wrong (real bug)     0
wall time            4.994 s
throughput (correct) 25630 ops/sec
latency  min 87us · p50 2372us · p90 3127us · p99 6589us · max 70659us
flood (256 conns < 512 ceiling): 32 probes answered, 0 refused, recovered ✓
```

Under the realistic shipped configuration, **every one of 128,000 concurrent
reads succeeded**. The connect-retry-with-backoff added in 0.6.0 absorbed the
listener-backlog blips that, before it, produced a handful of hard connect
errors.

### Run B — ceiling forced low (`SVAULT_STRESS_MAXCONN=64`)

```
correct              117151
refused (busy)       10823    (ceiling actively shedding load)
conn err (backlog)   26
wrong (real bug)     0
wall time            4.523 s
throughput (correct) 25900 ops/sec
latency  min 84us · p50 2326us · p90 3044us · p99 5292us · max 110006us
flood (256 conns > 64 ceiling): 0 answered, 32 refused busy, recovered ✓
```

With the ceiling set below the offered concurrency, the cap does its job: it
sheds ~8% of requests as clean `busy` refusals and the daemon stays correct and
recovers. This is the DoS bound (finding #8) working when tuned tight.

## Conclusions

- **Zero wrong values across every run.** The concurrency design is correct: a
  `Get` clones the 32-byte key and bumps `last_used` under the lock, then
  decrypts *outside* it, so parallel reads neither serialize nor race. Combined
  with the poison-recovery lock (finding #13), no single handler can corrupt
  shared state or take the daemon down.
- **The default ceiling (512) does not interfere with realistic load.** Even at
  64-way concurrency the shipped config refused nothing. Operators on small or
  shared hosts can lower `daemon.max_connections`; the cap still bounds runaway
  thread growth, as Run B shows.
- **Connect retry eliminated the transport-error tail** seen in pre-retry runs
  (~0.01% hard connect failures → 0 under the default config).
- **Throughput** held around **26k correct reads/sec** with a p99 near 5–7 ms,
  each read performing a full Argon2-free `open_with_key` + AES-256-GCM decrypt.

## Carry-forward

Re-run for the next `0.x.0` and record results in `stress/<version>.md`. Worth
adding later: a sustained-duration soak (memory-growth check) and a mixed
unlock/lock/get churn profile.
