# io_uring SQPOLL: experiment results (2026-06-13)

Follow-up to [`sqpoll-noise.md`](./sqpoll-noise.md) — runs the experiment matrix
recommended by the GPT-5 Codex review. Knob: `Config::io_backend`
(`IoBackend::{Uring { sqpoll_idle_ms }, Pwrite}`). These runs predate the #47
default flip — each backend is exercised explicitly here, and the default is now
`Pwrite` (chosen on the strength of these results). Harness:
`armdb/examples/sqpoll_matrix.rs`
(`cargo run --release -p armdb --features bench-internals --example sqpoll_matrix`).

- **Machine:** Ryzen 7 7700 (8 physical / 16 logical, Zen 4), Linux 6.8.0-117.
- **Governor:** `powersave` (absolute throughput is somewhat depressed, but every
  variant ran under the same governor, so the *relative* comparisons hold; the
  headline signals — CPU-ratio and poller-thread count — are governor-independent).
- **Build:** release. **Method:** 1 warmup + 3 measured reps, median; p99 from the
  pooled per-op latencies of the 3 measured reps; process **not** pinned.
- There is **no fsync axis**: `Config::enable_fsync` is a documented no-op for the
  Bitcask write path, so submission frequency is probed in Scenario 3 instead.

`CPU x` = process-CPU-seconds / wall-seconds (≈ number of cores kept busy).
`iou-sqp` = live `iou-sqp-*` kernel poller threads counted via `/proc/self/task`.

## Scenario 1 — single ConstTree (IoBackend × threads × shard_count)

```
          io  shards  threads    Mops/s    CPU x    p99 us   iou-sqp
 sqpoll-2000      16        1      0.29     14.9       5.1        16
 sqpoll-2000      16        8      0.44     14.9       5.3        16
 sqpoll-2000      16       16      0.46     14.9       6.7        16
   sqpoll-50      16        1      0.53      1.8       4.4        16
   sqpoll-50      16        8      2.05      8.5      18.0        16
   sqpoll-50      16       16      2.20     11.0      47.5        16
   no-sqpoll      16        1      0.56      1.0       4.0         0
   no-sqpoll      16        8      2.51      6.8      16.0         0
   no-sqpoll      16       16      2.84     10.1      41.8         0
      pwrite      16        1      0.56      1.0       4.3         0
      pwrite      16        8      2.55      6.8      15.7         0
      pwrite      16       16      2.85     10.2      42.2         0
 sqpoll-2000       8        1      0.50      9.0       4.9         8
 sqpoll-2000       8        8      1.53     12.7      32.8         8
   no-sqpoll       8        1      0.55      1.0       4.4         0
   no-sqpoll       8        8      1.73      5.5      31.3         0
      pwrite       8        1      0.50      1.0       5.1         0
      pwrite       8        8      1.76      5.5      30.8         0
```

- `sqpoll-2000` at `T=1`, 16 shards burns **CPU x ≈ 14.9** while delivering the
  *lowest* throughput (0.29 Mops/s) — 16 pollers spin even though one app thread
  is writing. This reproduces the original `sqpoll-noise.md` observation exactly.
- `pwrite` and `no-sqpoll` are statistically identical (CPU x = 1.0, 0 pollers, and
  the *highest* throughput) — the synchronous `submit_and_wait(1)` path gets nothing
  from io_uring here, so dropping the poller is pure win.
- Under load (`T=16`) SQPOLL is actively harmful: `pwrite` reaches 2.85 Mops/s vs
  `sqpoll-2000`'s 0.46 — the 16 pollers contend with the 16 app threads for the 8
  physical cores. `sqpoll-50` (short idle) sits in between.

## Scenario 2 — 1 vs N independent Engines (poller multiplier)

```
          io  collections shards/coll    Mops/s    CPU x   iou-sqp
 sqpoll-2000            1           8      3.21     11.2         8
 sqpoll-2000           10           8      1.46     14.9        80
   no-sqpoll            1           8      3.48      6.0         0
   no-sqpoll           10           8      7.10     14.0         0
      pwrite            1           8      3.37      6.1         0
      pwrite           10           8      7.35     14.0         0
```

- The `collection × shards` multiplier is **confirmed**: 10 Bitcask collections ×
  8 shards = **80 `iou-sqp` threads** for `sqpoll-2000` (vs 8 at one collection),
  exactly as GPT-5 predicted for the Armour `Db` case.
- The cost is severe: with 80 pollers thrashing 16 logical cores, `sqpoll-2000`
  throughput *drops* from 3.21 → 1.46 Mops/s as collections scale up, while
  `pwrite`/`no-sqpoll` *scale up* 3.37 → 7.35 Mops/s with **0** poller threads.
  At 10 collections `pwrite` is **~5× faster** than `sqpoll-2000`.

## Scenario 3 — flusher submission-frequency (armour::Db, low bg write rate)

```
          io    flusher          CPU/s   iou-sqp
 sqpoll-2000       true          8.014         8
 sqpoll-2000      false          3.518         8
      pwrite       true          0.002         0
      pwrite      false          0.000         0
```

- With a trickle of background writes and the 1 s flusher **on**, `sqpoll-2000`
  burns **8.0 CPU-seconds per wall-second** (all 8 pollers stay hot — each 1 s tick
  re-arms them inside the 2 s `sq_thread_idle`). Turning the flusher **off** roughly
  halves that to 3.5 (pollers get longer naps between the trickle's own flushes).
  So the default 1 s flusher meaningfully amplifies SQPOLL idle cost.
- `pwrite` is ~0 CPU/s either way — there is no poller to keep awake.

## Scenario 4 — ATTACH_WQ PoC

```
=== N=8 rings ===
  independent SQPOLL rings : 8 iou-sqp threads
  SQPOLL + ATTACH_WQ rings : 1 iou-sqp threads
  verdict: ATTACH_WQ COLLAPSES pollers
=== N=16 rings ===
  independent SQPOLL rings : 16 iou-sqp threads
  SQPOLL + ATTACH_WQ rings : 1 iou-sqp threads
  verdict: ATTACH_WQ COLLAPSES pollers
```

- On this kernel (6.8) `IORING_SETUP_ATTACH_WQ` **does collapse N SQPOLL pollers to
  a single shared `iou-sqp` thread** (8→1 and 16→1). This *refutes* the GPT-5
  caution that ATTACH_WQ might only share `io-wq` workers without removing the
  per-ring poller — measured here, one poller serves all attached rings.
- Caveat: this PoC only counts threads; it does not yet measure whether one shared
  poller keeps up with N rings' submission load under throughput. That is the
  follow-up before adopting ATTACH_WQ as *the* fix.

## Verdict

- **SQPOLL cost vs benefit:** for armdb's synchronous `submit_and_wait(1)` flush
  path, SQPOLL is all cost and no benefit. `pwrite` and `no-sqpoll` match or beat
  SQPOLL on throughput and p99 at **1/15th** the CPU at `T=1`, and SQPOLL actively
  loses under load and at scale.
- **Multiplier confirmed:** 8 → 80 `iou-sqp` threads from 1 → 10 collections, with
  throughput going the wrong way (3.21 → 1.46 Mops/s) while pwrite scales to 7.35.
- **Flusher keeps pollers hot:** the 1 s `Db` flusher holds idle CPU at ~8 CPU/s for
  SQPOLL even under a mere trickle of writes.
- **ATTACH_WQ collapses pollers** (8/16 → 1) on this kernel — a viable lever if
  SQPOLL must be retained, pending a throughput PoC of the shared poller.

**Recommendation (for the deferred follow-up PR):** make the default Bitcask flush
path `IoBackend::Pwrite` (equivalently io_uring **without** SQPOLL — they tie here),
and keep `Uring { sqpoll_idle_ms: Some(..) }` as an explicit opt-in for a future
*asynchronous* submission path that actually batches (today's `submit_and_wait` does
not). The `IoBackend` knob added in this branch is exactly that opt-in switch; this
PR does **not** flip the default.

> Reproduce: `cargo run --release -p armdb --features bench-internals --example sqpoll_matrix`
> (build in release — the example pins a literal hasher key, so no `APP_ID_KEY_*`
> env var is needed).
