# ADR 0013: QoS-priority sync fanout

**Status:** Proposed
**Date:** 2026-06-09
**Issue:** [peat-mesh#247](https://github.com/defenseunicorns/peat-mesh/issues/247)

---

## Context

Document convergence is fast in isolation (~8–13 ms) but, under sustained /
concurrent write load in the dual-C2 failover topology, a latency-sensitive
document's convergence degrades — intermittent multi-second-to-minute tails.
The degradation is **not** bandwidth-bound (unshaped links degrade too) and
**not** a correctness bug (everything eventually delivers). It is the last
thing between the rc.37 convergence numbers and a publishable matrix, and it
violates the determinism SLA (a busy C2 pushing many sitreps, or a relay
fanning to many platforms, must not delay a mission-critical alert behind the
backlog).

### What we measured (in-process, instrumented)

A faithful in-process reproduction now exists at
`tests/sync_fanout_under_load_repro.rs` (`#[ignore]`d until this lands). It
drives `AutomergeSyncCoordinator::sync_document_with_all_peers` serially for a
backlog of 24 large (64 KiB) Bulk-QoS documents (`sitreps`) followed by **one
small Critical-QoS document last** (`alerts`), and measures the critical doc's
**apply rank** on the receiver via `AutomergeStore::subscribe_to_observer_changes`
(fires on every applied put, in apply order — timing-independent, immune to the
`get()`-polling starvation that confounded an earlier attempt).

Result: the Critical-QoS document applies at **rank 24/25 — dead last.**

Two earlier hypotheses were measured and **falsified**:

1. *"Redundant per-peer `store.get` + `doc.save()` is the dominant cost."*
   Hoisting that work to once-per-document (kept — see Decision) left the
   number unchanged. It helps the heavy FullHistory per-peer Automerge-*clone*
   cost (the delta/history-depth scaling dimension) but is not the fix.
2. *"The single per-peer QUIC stream serializes a small frame behind bulk
   bytes; a frame-level priority lane fixes it."* A QoS+size priority lane was
   added to `SyncChannel` (kept — see Decision) and **the critical frame is
   correctly routed to it**, but the critical doc still applied dead last.
   Instrumentation showed why: when the writer reaches the critical frame, the
   bulk queue is already empty — the serial fanout driver is **paced by
   per-document prepare cost** (`load` + `save`), so the single QUIC writer
   drains in lockstep and **no queue ever builds for the lane to reorder.**

### Root cause

The head-of-line stall is **upstream of the channel.** A document's fanout does
not *start* until all prior documents' fanouts complete serially (the
`change_rx` → `await sync_document_with_all_peers` loop in
`AutomergeBackend::start_sync`, mirrored by `peat-node`'s relay
`sync_on_change`). A critical document enqueued behind a bulk backlog therefore
waits for the entire backlog regardless of any per-frame or per-byte
reordering downstream. **There is no priority signal at the level where the
ordering is actually decided: which document to fan out next.**

A prior naive fix (documented on #247) — spawn every change concurrently +
`join_all` — regressed 2.3 s → 7.25 s: it added contention on the single
per-peer channel and the `peer_states` lock and discarded the implicit
backpressure/ordering the serial loop provided. Any fix must preserve
per-(peer, channel) ordering and bounded concurrency.

## Decision

Introduce a **QoS-priority fanout queue** owned by the coordinator, draining to
the existing per-document fanout path:

1. **Priority queue, single drainer.** The coordinator owns a priority queue of
   pending fanout requests keyed by `QoSClass::for_collection(collection)`
   (`Critical > High > Normal > Low > Bulk`; `QoSClass` implements `Ord`), with
   FIFO ordering within a class (monotonic sequence tiebreak). A **single**
   worker task pops the highest-priority pending document and runs the existing
   prepare + per-peer fanout. Single-worker (not a pool) preserves the
   per-(peer, channel) ordering and natural backpressure the serial loop gave us
   — it is the serial loop, reordered by priority, not parallelized. This
   sidesteps the failed naive-concurrency shape.

2. **Coalescing.** Multiple pending requests for the same `doc_key` collapse to
   one (the latest local state is fanned once when the worker reaches it),
   bounding work under sustained same-document writes and complementing the
   existing per-(peer, doc) sync cooldown (#239).

3. **Non-blocking enqueue API.** A new coordinator method (working name
   `enqueue_fanout(doc_key)`) pushes onto the priority queue and returns
   immediately. `AutomergeBackend::start_sync`'s change loop calls it instead of
   `await`-ing `sync_document_with_all_peers`, so the driver loop drains its
   `change_rx` at memory speed and the priority queue — not arrival order —
   decides fanout order. `sync_document_with_all_peers` remains for direct /
   synchronous callers and as the worker's per-document primitive.

4. **Bounded, with priority-ordered shedding.** The queue has a single
   **absolute** bound (default `4096` pending entries; configurable). Absolute
   rather than per-class because coalescing (#2) already collapses same-document
   churn, so the queue tracks *distinct in-flight documents*, not write rate —
   4096 distinct hot documents is generous for tactical meshes and bounds memory
   at roughly the doc-key strings + priority metadata (kilobytes). Per-class
   *bounds* are rejected for the initial slice (they complicate backpressure and
   the shed order already protects high-priority traffic); per-class **drop
   counters** are exposed instead for observability.

   **Overflow / drop semantics.** Enqueue is non-blocking and **never errors
   back to the change loop** — returning `Err` there is exactly the
   peat#864 shape (the auto-sync pusher swallows the error and the change is
   silently lost from the outbound pipeline with no retry). Instead, when the
   queue is full the **lowest-priority pending entry is evicted** to admit the
   new one; if the incoming entry is itself the lowest priority (and not newer
   than the tail), the incoming entry is the one dropped. Every eviction is
   **counted (per-class metric) and `tracing::warn`-logged**. This is safe:
   shedding only happens under sustained overload the bound exists to cap, it
   sheds Bulk/Low before anything mission-relevant, and an evicted fanout is
   recovered by the next local change to that document or the periodic /
   reconnect re-sync — consistent with the at-most-once-eventually delivery
   already documented for `StateSnapshot`/`Tombstone` on the persistent channel.
   A Critical/High entry is therefore only ever evicted if the *entire* queue is
   Critical/High, i.e. the mesh is genuinely Critical-saturated — a load-shedding
   condition, not a priority-inversion bug.

### Retained, complementary pieces (already implemented on the branch)

- **Per-document load/serialize once** (`PreparedDoc` in `automerge_sync.rs`):
  `store.get` (Automerge clone) + `doc.save()` happen once per fanout and are
  shared across peers, instead of once per peer. Cuts the per-document constant
  the worker pays — directly relevant because that constant is what paces the
  fanout.
- **Frame-level QoS+size priority lane** (`SyncChannel`, `automerge_sync.rs`):
  two send lanes per peer (priority drained ahead of bulk; small frames or
  Critical/High collections take the priority lane). Necessary-but-insufficient
  alone, but it closes the residual within-channel window: once the priority
  worker hands a critical frame to a peer whose writer is mid-transfer of a
  bulk frame, the critical frame still preempts the *remaining* bulk frames
  rather than queuing behind them.

## Consequences

- A Critical/High-QoS document converges near the per-document floor regardless
  of how much lower-priority traffic is queued — the determinism SLA shape.
  Same-QoS-class load remains throughput-bound (N documents through one stream
  genuinely take N transfers; that is the transfer floor, not a bug).
- Fanout order is no longer arrival order. For CRDTs this is safe (merges are
  commutative; per-document delivery is still ordered per (peer, channel)).
- Delivery semantics shift from "awaited-enqueued" to "scheduled"; the worker
  owns ret/ error handling exactly as the per-document path does today.

## Cross-repo note

`peat-node`'s relay `sync_on_change` loop is the other site of this serial
fanout. This ADR fixes the peat-mesh-owned driver (`start_sync`). Making a
*relay* prioritize re-fanned inbound changes requires the analogous change in
peat-node (tracked separately, linked from #247) — peat-mesh cannot reorder a
consumer's drive loop, only its own.

## Alternatives considered

- **Frame-level priority lane only** — measured insufficient (critical doc
  still rank 24/25): it reorders frames already queued in the channel, but the
  serial driver never builds a queue.
- **Naive concurrent fanout (spawn-per-change + `join_all`)** — already tried
  upstream; regressed 2.3 s → 7.25 s via single-channel + `peer_states`
  contention and loss of backpressure. Rejected.
- **Worker pool (N drainers)** — reintroduces the cross-peer channel + lock
  contention the single drainer avoids; the bottleneck is ordering, not fanout
  parallelism. Rejected for the initial slice.
- **Interleaving / chunking large transfers, or a separate QUIC stream for
  bulk** — addresses the within-stream byte-HOL for the same-QoS case, a larger
  change to the ADR-063 persistent-stream contract. Deferred; the priority
  queue + frame lane cover the cross-QoS case that the determinism SLA is about.

## Verification

- `tests/sync_fanout_under_load_repro.rs` (un-`ignore`d once implemented):
  Critical-QoS doc applied at rank ≤ 3 behind a 24-doc Bulk backlog.
- Existing suites: `cargo test --features automerge-backend`,
  `cargo test --features node`, plus the storage unit tests pinning the
  defer-on-cooldown (peat#864) and absent-doc breaker (#243) invariants.
