# gigastt v0.4 Feature Design

Three features delivered as sequential sub-projects: flexible sample rate, HTTP/REST API with SSE, and speaker diarization.

## Delivery Order

1. **Flexible Sample Rate** — unblocks integration with Meeting-LLM (8kHz) and call-translator (24kHz)
2. **HTTP/REST API + SSE** — opens simple integration without WebSocket
3. **Speaker Diarization** — adds speaker identification for meeting transcription

Each feature gets its own plan -> implement -> test cycle. Features are independent but share protocol evolution.

---

## Feature 1: Flexible Sample Rate

### Goal

Accept audio at any common sample rate (8kHz, 16kHz, 24kHz, 44.1kHz, 48kHz) via WebSocket. Resample to 16kHz server-side using rubato (polyphase FIR). Backward compatible: clients not sending Configure get 48kHz default.

### Protocol Changes

**New `ClientMessage::Configure`:**
```rust
Configure { sample_rate: u32 }
```
- Sent before first audio chunk
- Validated against supported rates: `[8000, 16000, 24000, 44100, 48000]`
- Error response on invalid rate
- If not sent: default 48kHz (backward compatible)

**`ServerMessage::Ready` extension:**
```rust
Ready {
    model: String,
    sample_rate: u32,           // default expected rate (48000)
    version: String,
    supported_rates: Vec<u32>,  // NEW: [8000, 16000, 24000, 44100, 48000]
}
```
`supported_rates` uses `#[serde(skip_serializing_if = "Vec::is_empty")]` for backward compatibility.

### Resampling

Replace linear interpolation in `audio::resample()` with rubato `SincFixedIn`:
- rubato already in Cargo.toml (unused)
- Polyphase FIR quality vs current linear interpolation
- New `Resampler` struct wrapping rubato instance, created per-connection
- Stored in `StreamingState` as `Option<Resampler>` (None when input is already 16kHz)

### Server Logic

1. After sending Ready, accept Configure or Binary frame
2. If first frame is Binary: assume default 48kHz, create resampler
3. If Configure: validate rate, create resampler (or None for 16kHz), acknowledge
4. All subsequent Binary frames resampled through rubato to 16kHz
5. `process_chunk()` receives 16kHz — no changes to inference pipeline

### Files Changed

| File | Change |
|------|--------|
| `src/protocol/mod.rs` | `ClientMessage::Configure`, `Ready` + `supported_rates` |
| `src/inference/audio.rs` | Replace `resample()` with rubato `Resampler` struct |
| `src/server/mod.rs` | Configure handling, per-connection resampler |
| `src/inference/mod.rs` | `StreamingState` + `Option<Resampler>` field |

### Acceptance Criteria

- [ ] `Configure { sample_rate: 8000 }` accepted, audio resampled correctly
- [ ] All 5 supported rates work: 8000, 16000, 24000, 44100, 48000
- [ ] No Configure message → default 48kHz (backward compatible)
- [ ] Invalid rate (e.g., 7000) returns Error message
- [ ] Ready message includes `supported_rates` field
- [ ] rubato resampler produces better quality than linear interpolation
- [ ] Existing tests pass without modification
- [ ] New unit tests for each supported rate
- [ ] Integration test: Configure + 8kHz audio stream

---

## Feature 2: HTTP/REST API + SSE

### Goal

Add HTTP endpoints for file-based transcription alongside existing WebSocket streaming. Includes SSE for streaming transcription over HTTP. Single port serves both HTTP and WebSocket.

### Dependencies

Add to Cargo.toml:
- `axum` — HTTP framework (tokio-native)
- `axum-extra` — multipart form support
- `tower-http` — CORS, body limits

### Endpoints

| Method | Path | Content-Type | Response |
|--------|------|-------------|----------|
| `GET` | `/health` | — | `{"status":"ok","model":"gigaam-v3","version":"0.4.0"}` |
| `POST` | `/v1/transcribe` | `multipart/form-data` or `audio/*` | `{"text":"...","words":[...],"duration":5.2}` |
| `POST` | `/v1/transcribe/stream` | `multipart/form-data` or `audio/*` | SSE: `data: {"type":"partial",...}` events |
| `GET` | `/ws` | WebSocket upgrade | Existing WebSocket protocol |

### POST /v1/transcribe

- Accept file upload (multipart field `file`) or raw audio body
- Supported formats: WAV, MP3, M4A/AAC, OGG, FLAC (same as CLI `transcribe`)
- Max file size: 50MB
- Max duration: 10 minutes (same limit as CLI)
- Response: JSON with full transcript, word-level timing, duration
- Reuses `engine.transcribe_file()` via temp file

### POST /v1/transcribe/stream (SSE)

- Same input as `/v1/transcribe`
- Response: `Content-Type: text/event-stream`
- Events mirror WebSocket protocol: `partial` and `final` types
- File audio chunked through `engine.process_chunk()` for incremental results
- Final event: `data: {"type":"final","text":"full transcript","words":[...]}`

### Server Architecture

Migrate from raw tokio-tungstenite to axum:
- axum natively supports WebSocket upgrade (`axum::extract::ws`)
- Single port serves HTTP routes + WebSocket at `/ws`
- WebSocket at root `/` preserved for backward compatibility (redirect or dual handler)
- Semaphore shared between HTTP and WebSocket handlers (max 4 concurrent)

### Files Changed

| File | Change |
|------|--------|
| `Cargo.toml` | + `axum`, `axum-extra`, `tower-http` |
| `src/server/mod.rs` | Refactor: axum router, WS upgrade, shared state |
| `src/server/http.rs` | **New**: health, transcribe, transcribe/stream handlers |
| `src/protocol/mod.rs` | `TranscribeResponse` struct for HTTP |
| `src/main.rs` | Update serve command to use new server |

### Acceptance Criteria

- [ ] `GET /health` returns 200 with model info
- [ ] `POST /v1/transcribe` with WAV file returns JSON transcript
- [ ] `POST /v1/transcribe` with MP3/M4A/OGG/FLAC works
- [ ] `POST /v1/transcribe/stream` returns SSE events (partial + final)
- [ ] WebSocket still works at `/ws` (backward compatible)
- [ ] File size limit enforced (50MB)
- [ ] Duration limit enforced (10 min)
- [ ] Concurrent request limiting (shared semaphore)
- [ ] Health check works for Docker HEALTHCHECK
- [ ] Existing WebSocket integration tests pass

---

## Feature 3: Speaker Diarization

### Goal

Identify and label speakers in transcribed audio using a separate speaker embedding model. Optional feature behind `--features diarization` compile flag.

### Model

- **wespeaker-voxceleb-resnet34** ONNX (~26MB) or ECAPA-TDNN from SpeechBrain (~30MB)
- Downloaded separately: `gigastt download --diarization`
- Optional: if model not present, diarization unavailable (graceful degradation)

### Pipeline

```
Audio chunk (16kHz)
  ├──> GigaAM encoder → decoder → text + words (existing)
  └──> Speaker encoder → 192-dim embedding
       └──> Online clustering → speaker_id (u32)
```

- Speaker encoder runs on ~1.5s audio segments
- Online incremental clustering (cosine similarity + threshold-based assignment)
- New speaker detected when similarity < threshold → new ID assigned
- Speaker ID attached to each word in `WordInfo`

### Protocol Changes

**`WordInfo` extension:**
```rust
pub struct WordInfo {
    pub word: String,
    pub start: f64,
    pub end: f64,
    pub confidence: f32,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub speaker: Option<u32>,  // NEW
}
```

**`ServerMessage::Ready` extension:**
```rust
diarization: bool  // NEW: whether diarization model is loaded
```

**`ClientMessage::Configure` extension:**
```rust
Configure {
    sample_rate: Option<u32>,
    diarization: Option<bool>,  // NEW: enable/disable per session
}
```

### Compile-Time Feature

```toml
[features]
diarization = []  # gates diarization module compilation
```

All diarization code behind `#[cfg(feature = "diarization")]`. Default builds exclude it entirely.

### Files Changed

| File | Change |
|------|--------|
| `Cargo.toml` | `diarization` feature |
| `src/inference/diarization.rs` | **New**: speaker encoder, embedding extraction, online clustering |
| `src/inference/mod.rs` | Integration with process_chunk (optional) |
| `src/protocol/mod.rs` | `speaker` in WordInfo, `diarization` in Ready/Configure |
| `src/model/mod.rs` | Speaker model download (`--diarization` flag) |
| `src/main.rs` | `download --diarization` CLI option |

### Acceptance Criteria

- [ ] Speaker model downloads via `gigastt download --diarization`
- [ ] Without model: diarization unavailable, no errors
- [ ] With model + `Configure { diarization: true }`: speaker IDs assigned to words
- [ ] 2-speaker test scenario: correctly assigns distinct speaker IDs
- [ ] 3-speaker test scenario: identifies 3 speakers
- [ ] `speaker` field absent when diarization disabled (backward compatible)
- [ ] `--features diarization` compiles; default build excludes diarization code
- [ ] Existing tests pass without diarization feature
- [ ] New unit tests: embedding extraction, online clustering
- [ ] Diarization adds < 50ms latency per chunk

---

## Cross-Cutting Concerns

### Protocol Versioning

Protocol version stays "1.0" — all changes are additive:
- New `supported_rates` field in Ready (skip_serializing_if empty)
- New `speaker` field in WordInfo (skip_serializing_if None)
- New `diarization` field in Ready (skip_serializing_if false)
- New `Configure` message (unknown messages from old clients ignored)

### Backward Compatibility

- Clients not sending Configure get 48kHz default
- Clients ignoring `supported_rates`/`diarization` in Ready work fine
- HTTP endpoints are new — no existing clients affected
- `speaker: null` in JSON omitted entirely (skip_serializing_if)

### Version Bump

After all three features: bump to v0.4.0.
