# ADR-0034: `tract-onnx` as the Speaker-Embedding Runtime

## Status

✅ Accepted (2026-05-14)

## Context

[ADR-0033](adr-0033.md) picked `candle` as the production ASR runtime
for [#802](https://github.com/rust-works/omni-dev/issues/802), closing
the no-C++ question for transcription. The
[2026-05-14 update](https://github.com/rust-works/omni-dev/issues/805)
to [#805](https://github.com/rust-works/omni-dev/issues/805) noted that
**speaker embedding is a separate runtime question**: `candle` does not
ship a first-party speaker-embedding model, and reusing the existing
runtime is only attractive if a HuggingFace embedding model is loadable
under `candle` without re-implementing the architecture by hand.

Spike #805 (branch `issue-805-spike-tract-speaker`, captured in
`SPIKE.md`) prototyped the second option from the issue's preference order —
**`tract-onnx`** running sherpa-onnx's pre-trained wespeaker ONNX
model — against the Day-1 stop rule. The spike also audited why the
preferred option (`candle` + HF model) was not viable in half a day.

The acceptance bars and `tract-onnx`'s measured results on the
committed fixture [tests/fixtures/voice/two_speakers.wav](../../tests/fixtures/voice/two_speakers.wav)
(24.5 s, 16 kHz mono 16-bit PCM, two LibriVox readers):

| Bar | Result |
| --- | --- |
| Separability — within-speaker (gate ≥ 0.70) | min/mean = **0.90 / 0.91** |
| Separability — cross-speaker (gate ≤ 0.40) | max/mean = **0.09 / 0.07** |
| Model load (mmap + tract optimisation, gate ≤ 1.00 s) | ~80 ms |
| Inference latency (warm, 6 s window, gate ≤ 0.50 s) | ~200 ms/window |
| Determinism (3 reruns of one window) | bit-equal embeddings |
| `cargo tree \| grep -iE 'cmake\|cc-build\|c\+\+'` | empty — passes literal acceptance grep |
| Cold build time | 3 m 05 s |
| Binary size (release, LTO=thin, codegen-units=1) | 14 MiB |

`tract-onnx` passed every measured bar with substantial margin. Per the
Day-1 rule, no fall-back candidate was prototyped; SPIKE.md's candidate
audit documents the skip rationale for `candle` + HF model — see
**Alternatives considered** below.

## Alternatives considered

The Day-1 stop rule prevented prototyping fall-backs once `tract-onnx`
cleared every gate. The audit considered the following options and
recorded skip rationale; none change the decision, but they are
documented for traceability and so future consolidation work has a
starting point.

### `candle` + HuggingFace speaker-embedding model (rejected)

- **No first-party reference implementation.**
  [candle-transformers/src/models](https://github.com/huggingface/candle/tree/main/candle-transformers/src/models)
  ships ~100+ models (Llama, Mistral, Whisper, Mimi, Encodec, CLIP,
  …) but none for speaker verification — no wespeaker, WavLM-SV,
  ECAPA-TDNN, x-vector, or TitaNet. Audio coverage is biased toward
  ASR and codecs.
- **Available weights aren't candle-friendly.**
  [`Wespeaker/wespeaker-voxceleb-resnet34-LM`](https://huggingface.co/Wespeaker/wespeaker-voxceleb-resnet34-LM/tree/main)
  ships the ONNX (26.5 MB) plus an `avg_model` raw pickle (45.1 MB) —
  no safetensors and no Transformers-style `pytorch_model.bin`.
  [`microsoft/wavlm-base-plus-sv`](https://huggingface.co/microsoft/wavlm-base-plus-sv/tree/main)
  ships only `pytorch_model.bin` (405 MB) — no safetensors, no ONNX.
  Either path forces a reimplementation rather than a load.
- **Hand-port cost is real.** wespeaker ResNet34-TSTP-emb256 is
  6.63 M params / 4.55 GFLOPs (fbank80 input); the architecture is a
  standard 2D ResNet-34 trunk + temporal statistics pooling + an FC
  head, well within candle's expressive range. But re-implementing it
  (including the fbank80 feature extractor) with bit-equivalence to
  the reference is a 2–3 day exercise with silent-numerical-drift
  risk and no golden output to compare against until the ONNX is
  loaded — circular.

### `candle-onnx` (deferred, not rejected outright)

[`candle-onnx`](https://crates.io/crates/candle-onnx) lets candle load
ONNX models directly, which would in principle have avoided introducing
a second runtime. Not pursued in this spike for two reasons:

- **Op coverage is incomplete.** Open issues such as
  [candle#2876](https://github.com/huggingface/candle/issues/2876)
  flag commonly-needed ONNX ops (e.g. `Resize`) as unsupported.
  Verifying every op in the wespeaker ResNet-34 + TSTP graph runs
  through `candle-onnx` is itself a half-day exercise — outside the
  Day-1 budget.
- **Day-1 stop rule.** `tract-onnx` had already cleared every
  acceptance gate with substantial margin; the spike stopped there.

**Future consolidation.** If `candle-onnx` op coverage closes the gaps
the wespeaker graph hits, the production code can swap
`WespeakerEmbedder`'s tract backend for a candle-onnx-backed one with
no public API change. Worth a follow-up spike if the two-runtime cost
ever becomes a maintenance burden, or if a candle-transformers
reference impl for a speaker model lands upstream.

### `native-pyannote-rs` (Burn-based, skipped)

[`native-pyannote-rs`](https://crates.io/crates/native-pyannote-rs) is
a pure-Rust pipeline using **Burn + ndarray + kaldi-native-fbank**
that ships a working wespeaker-resnet34 speaker-identification path
today. Burn was ranked last in #805's preference order ("less mature
for audio"), but a working production-pipeline-shaped precedent
already exists in the ecosystem. Not pursued because:

- Adding Burn is a heavier dep-graph change than adding tract — Burn
  is a much larger framework with its own model-definition surface
  and a multi-backend abstraction.
- The Day-1 winner (`tract-onnx`) had already cleared every gate.

Recorded here so future runtime-consolidation reviews have the
precedent.

### Other ONNX runtimes considered

- **[`rten`](https://github.com/robertknight/rten)** — pure-Rust,
  actively maintained, runs ResNet-50 in the README. Plausible
  substitute for tract; skipped at Day-1 in favour of tract's larger
  production deployment record (Sonos on-device speech).
- **[`wonnx`](https://github.com/webonnx/wonnx)** — effectively
  discontinued (no release in 12+ months); correctly excluded.
- **[`ort`](https://github.com/pykeio/ort)** — Rust bindings to
  Microsoft's C++ ONNX Runtime; excluded by the no-C++ constraint.

### Alternative speaker models

ECAPA-TDNN
([`Wespeaker/wespeaker-ecapa-tdnn512-LM`](https://huggingface.co/Wespeaker/wespeaker-ecapa-tdnn512-LM),
EER 0.728 %) and 3D-Speaker CAM++ / ERes2NetV2
([modelscope/3D-Speaker](https://github.com/modelscope/3D-Speaker))
are near-equivalent-accuracy ONNX exports that would load under
`tract-onnx` without architectural changes. Not evaluated in the
spike; flagged as the obvious A/B candidates if
`wespeaker_en_voxceleb_resnet34_LM` ever underperforms on a
non-English fixture or in field conditions.

## Decision

The production speaker-embedding backend uses `tract-onnx` v0.21,
loading [`wespeaker_en_voxceleb_resnet34_LM.onnx`](https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/wespeaker_en_voxceleb_resnet34_LM.onnx)
(26.5 MB, SHA-256 `e9848563da86f263117134dfd7ad63c92355b37de492b55e325400c9d9c39012`)
from the sherpa-onnx
[`speaker-recongition-models`](https://github.com/k2-fsa/sherpa-onnx/releases/tag/speaker-recongition-models)
release.

The embedder is implemented as a new `WespeakerEmbedder` struct in
`src/voice/speaker.rs`. Its embedding API is a pure function:

```rust
impl WespeakerEmbedder {
    pub fn new(model_path: &Path) -> Result<Self>;
    pub fn embed(&self, pcm: &[i16]) -> Result<[f32; 256]>;
}
```

`embed` takes 16 kHz mono `i16` PCM (same shape as
[`AudioInput`](../../src/voice/transcriber.rs)), computes 80-dim Kaldi-style
FBANK features (25 ms / 10 ms framing, pre-emphasis 0.97, Hamming window,
mel scale `1127·ln(1 + f/700)`, cepstral mean normalisation across the
window), runs the ONNX graph, and returns an L2-normalised 256-dim
embedding.

The model is fetched at `voice install-model` time, not at
`voice enroll` / `voice transcribe` time. Path resolution lives in the
existing `src/voice/models.rs` after a small refactor (this ADR doesn't
mandate the exact shape; the SPIKE.md sketches the `ModelSpec` shape
the production PR will pick).

Storage convention (parallel to ADR-0033):

1. `--speaker-model <path>` on `voice enroll` / `voice transcribe` /
   `voice install-model --speaker-dest`.
2. `OMNI_DEV_VOICE_SPEAKER_MODEL` env var.
3. Default: `~/.omni-dev/voice/models/wespeaker-en-voxceleb-resnet34-LM/wespeaker_en_voxceleb_resnet34_LM.onnx`.

Enrolled speaker embeddings are persisted to
`~/.omni-dev/voice/speakers/<name>.json` per the original #805 spec.

`voice transcribe --speaker <name> [--threshold 0.5]` is a separate
composable layer on top of the existing `Transcriber` event stream —
**the `Transcriber` trait is not extended**. The filter takes the
source PCM alongside the events; for each `Final`, it embeds the
corresponding PCM window and emits the event only when
`cosine(window, enrolled) ≥ threshold`. Non-matching segments are
dropped (v1 scope: filter, not tag).

## Consequences

### Positive

- **Speaker embedding ships in half a day.** The spike resolved the
  runtime question with substantial margin on every gate; the
  remaining work is plumbing (CLI, persistence, models.rs refactor)
  rather than algorithmic risk.
- **No re-implementation tax.** wespeaker's ONNX graph is a complete
  trained model; `tract-onnx` runs it without architecture-specific
  glue, in contrast with the rejected option of hand-porting
  wespeaker-resnet34-SE forward pass into candle.
- **Same install convention as Whisper.** The
  [`src/voice/models.rs`](../../src/voice/models.rs) registry pattern
  established by ADR-0033 generalises naturally — one `ModelSpec` per
  model variant, registry-driven download, atomic install. Users see
  one consistent `omni-dev voice install-model` UX for both ASR and
  speaker embedding.
- **Trait surface untouched.** Speaker filtering is a separate layer
  over the existing event stream — no `Transcriber` trait change, no
  `MockTranscriber` impact, no breaking change to #802's contract.
- **Excellent separability headroom.** Within-speaker mean 0.91,
  cross-speaker mean 0.07 → ~0.84 separation. The proposed
  `--threshold 0.5` default has ≥ 0.4 margin on both sides; users can
  tune up or down without re-enrolling.

### Negative

- **A second pure-Rust inference runtime.** `candle` (ASR) and
  `tract-onnx` (speaker embedding) now coexist in the production
  binary. The combined compiled-deps story is "C toolchain required;
  no C++/cmake; one ONNX runtime + one safetensors runtime." Binary
  size impact is real but acceptable — measure in the production PR.
- **`cc`-via-`tract-linalg` build-dep.** Mirrors candle's
  `onig_sys`/`ring` caveat from ADR-0033. `tract-linalg` uses `cc`
  to compile platform-conditional SIMD assembly for vectorised
  matmul. The compiled output is **assembly**, not C++; no `cmake`,
  no `libstdc++`. The literal acceptance grep
  (`cmake|cc-build|c\+\+`) remains empty. A C toolchain is already
  required by the project (per ADR-0033's `onig_sys` and `ring`);
  this ADR adds no new toolchain requirement.
- **Cold build cost.** `tract-onnx` adds ~3 minutes to a cold
  `cargo build --release` of the spike crate. The production
  integration shares the dep graph with candle's tract-data /
  tract-linalg subset (both runtimes depend on tract-linalg
  transitively), so the actual incremental cost in the main binary
  will be lower than the spike's standalone 3 m 05 s. Incremental
  rebuilds are unaffected.
- **English-only model for v1.** `wespeaker_en_voxceleb_resnet34_LM`
  was trained on VoxCeleb (English-dominant). Voices in other
  languages should still embed sensibly (the model is speaker-
  identity-focused, not language-focused), but separability has not
  been measured on non-English fixtures. Multilingual variants
  (`wespeaker_zh_cnceleb_resnet34_LM`, 3D-Speaker CAM++ zh/en) are
  drop-in replacements in the same release; adding a `--variant`
  flag to `voice install-model` is an additive change later.

### Neutral

- **Embedding dimension is 256.** Smaller than ECAPA-TDNN-256 (192),
  comparable to most production wespeaker variants. Persistence
  files are ~3 KB each (256 × 8-byte f64 JSON + envelope).
- **Speaker filter dropped-segment behaviour is not configurable
  in v1.** The issue spec explicitly chose filter-not-tag for v1;
  multi-speaker tagging is deferred. The filter's threshold *is*
  configurable from the CLI.

## Out of Scope (Tracked Elsewhere)

- **Streaming speaker locking** — current design runs the embedder
  on completed `Final` segments. Streaming refinement (rolling
  embedding adaptation, partial-event speaker tagging) is part of
  [#806](https://github.com/rust-works/omni-dev/issues/806).
- **Multi-speaker diarisation** — v1 is single-speaker locking only.
- **Multilingual speaker models** — wespeaker zh / 3D-Speaker CAM++
  are drop-in; add a `--variant` flag to `voice install-model`
  later. No `Transcriber` or `WespeakerEmbedder` change needed.
- **`voice listen` (#807)** speaker locking — composes
  `WespeakerEmbedder` + the streaming pipeline; not in #805 scope.

## Coverage expectations

The seam this ADR establishes has the same shape of by-design
coverage gap as [ADR-0033](adr-0033.md) for candle and
[ADR-0031](adr-0031.md) for cpal. Codecov's patch-coverage number on
the production PR will appear low (~70 % overall, ~12 % on
`transcribe.rs`, ~61 % on `enroll.rs`) because the load-bearing code
paths intentionally cannot run in CI without staging the wespeaker
ONNX model on the runner. The following gaps are **by design** and
should not be chased with additional tests:

| Surface | Why uncovered |
|---|---|
| `WespeakerEmbedder::new` (ONNX parse + optimise + runnable plan) | Needs the staged wespeaker ONNX on disk; the missing-file error path *is* covered. |
| `WespeakerEmbedder::embed` (FBANK + ONNX run) | Needs the real model loaded. |
| `SpeakerFilter::load` and `SpeakerFilter::transform` Final-event arms in `src/cli/voice/transcribe.rs` | Same — the embedding call requires the model. The non-Final passthrough arms are covered by the `#[ignore]`-gated `voice_enroll_speaker_test`. |
| `EnrollCommand::execute` (cpal + run_capture + embed) | Needs a microphone (per [ADR-0031](adr-0031.md)) *and* the wespeaker model. The clap-parse layer, format-summary helpers, and read-WAV format validation *are* covered. |
| `InstallModelCommand::download_release_asset` (ureq GET + SHA-256 verify + atomic install) | Hits the network; coupling CI to the sherpa-onnx GitHub release is a hazard. The idempotent-skip and SHA-mismatch error paths are reachable via staged tempdir fixtures and *are* covered. |
| `InstallModelCommand::download_hf_hub` | Same — network-bound, mirrors the by-design gap from ADR-0033. |
| `Variant::SpeakerWespeakerEn` arm dispatch in `InstallModelCommand` | Same — the dispatch logic is covered (`variant_spec_returns_correct_spec`); the actual download arm is the gap. |

The local `#[ignore]`-gated integration test
[`tests/voice_enroll_speaker_test.rs`](../../tests/voice_enroll_speaker_test.rs)
exercises `WespeakerEmbedder::embed` end-to-end against
[`tests/fixtures/voice/two_speakers.wav`](../../tests/fixtures/voice/two_speakers.wav)
after `omni-dev voice install-model --variant speaker-wespeaker-en`,
asserting the same spike-aligned separability gates ADR-0034 was
accepted on (within-speaker ≥ 0.70, cross-speaker ≤ 0.40).

The FBANK feature extractor itself (`src/voice/features.rs`) is
covered by in-process unit tests — filterbank dimensionality,
non-negativity, monotonic centre-frequency ordering, frame-count
formula, CMN-zeros-mean-per-bin, hz/mel round-trip — designed to
catch silent drift in the DSP code without needing the model.

All *other* code paths in `src/voice/{features,paths,speaker}.rs`,
`src/cli/voice/{install_model,enroll,transcribe}.rs`, and
`src/voice/models.rs` have unit tests driven by in-process fixtures
(`TempDir`, staged-but-empty model dirs, in-memory CLI literals).
When a coverage gap appears in those files that isn't on the list
above, the right response is to add a test, not to extend this
exception list.

## References

- `SPIKE.md` on branch `issue-805-spike-tract-speaker` — measured
  metrics, candidate audit, reproduction.
- Issue [#805](https://github.com/rust-works/omni-dev/issues/805) —
  voice enrolment and speaker locking.
- Issue [#799](https://github.com/rust-works/omni-dev/issues/799) —
  umbrella; the 2026-05-14 update set the runtime preference order.
- [ADR-0033](adr-0033.md) — `candle` as the production ASR runtime
  (sibling decision; precedent for the install convention).
- sherpa-onnx [speaker-recongition-models](https://github.com/k2-fsa/sherpa-onnx/releases/tag/speaker-recongition-models)
  release — model source.
- [tract-onnx](https://github.com/sonos/tract) — pure-Rust ONNX
  runtime, ~85% op coverage in the relevant subset (Conv, BatchNorm,
  statistical pooling); resnet-family models load without
  modification.
- [candle-transformers models directory](https://github.com/huggingface/candle/tree/main/candle-transformers/src/models)
  — confirmed absence of any speaker-verification model.
- [candle-onnx](https://crates.io/crates/candle-onnx) +
  [candle#2876](https://github.com/huggingface/candle/issues/2876) —
  candle's ONNX loader plus an example of the op-coverage gap that
  motivates the deferral.
- [Wespeaker/wespeaker-voxceleb-resnet34-LM HF files](https://huggingface.co/Wespeaker/wespeaker-voxceleb-resnet34-LM/tree/main)
  — actual HuggingFace artefacts (ONNX + `avg_model` pickle; no
  safetensors).
- [microsoft/wavlm-base-plus-sv HF files](https://huggingface.co/microsoft/wavlm-base-plus-sv/tree/main)
  — `pytorch_model.bin` only.
- [pyannote/wespeaker-voxceleb-resnet34-LM](https://huggingface.co/pyannote/wespeaker-voxceleb-resnet34-LM)
  and [thewh1teagle/pyannote-rs](https://github.com/thewh1teagle/pyannote-rs)
  — established pyannote precedent for the same ONNX model.
- [native-pyannote-rs](https://crates.io/crates/native-pyannote-rs) —
  Burn-based pure-Rust precedent for a wespeaker-resnet34 pipeline;
  recorded for future consolidation reviews.
- wespeaker
  [voxceleb/v2 README](https://github.com/wenet-e2e/wespeaker/blob/master/examples/voxceleb/v2/README.md)
  — published EER (0.723 % on VoxCeleb1-O-clean) and architecture
  parameters (6.63 M params, fbank80 input).
- Alternative speaker models:
  [Wespeaker/wespeaker-ecapa-tdnn512-LM](https://huggingface.co/Wespeaker/wespeaker-ecapa-tdnn512-LM),
  [modelscope/3D-Speaker](https://github.com/modelscope/3D-Speaker).
