# Implementation Plan: Gemma 4 Local Inference (Opt-In Direct)

**Branch**: `041-adapter-gemma4-local` | **Date**: 2026-04-04 | **Spec**: [spec.md](spec.md)
**Input**: Feature specification from `/specs/041-adapter-gemma4-local/spec.md`

## Summary

Add direct in-process Gemma 4 inference to the `swink-agent-local-llm` crate, making Gemma 4 E2B the default local model (replacing SmolLM3-3B). This requires upgrading mistral.rs from 0.7 to 0.8, adding a `MultimodalModelBuilder` code path for Gemma 4 architecture, implementing a stateful cross-chunk thinking parser for Gemma 4's `<|channel>thought` delimiter format, and injecting the `<|think|>` control token for direct inference thinking mode. All changes are feature-gated behind `gemma4` until upstream stability is confirmed. The NaN logits bug (mistral.rs #2051) was reported against MoE variants (26B) with BF16/UQFF quantization — E2B (dense architecture) with Q4_K_M GGUF is likely unaffected. Implementation proceeds for E2B with a live validation gate after US1. E4B/26B (MoE) presets are deferred until the upstream fix ships.

## Technical Context

**Language/Version**: Rust 1.88, edition 2024  
**Primary Dependencies**: `mistralrs` 0.8+ (upgrade from 0.7), `hf-hub` 0.5, `tokio`, `futures`, `serde_json`, `tracing`  
**Storage**: Local filesystem — model weights cached in `~/.cache/huggingface/hub/` (managed by `hf-hub`)  
**Testing**: `cargo test` (unit), `cargo test --test local_live -- --ignored` (live with model download)  
**Target Platform**: macOS (Metal), Linux (CUDA/CPU), Windows (CPU)  
**Project Type**: Library crate (`swink-agent-local-llm`) within the `swink-agent` workspace  
**Performance Goals**: Streaming token-by-token inference; throughput depends on hardware (not a code-level target)  
**Constraints**: NaN logits bug (#2051) affects MoE variants with BF16/UQFF — E2B (dense, Q4_K_M GGUF) can proceed with live validation gate after US1. E4B/26B deferred until upstream MoE fix. All Gemma 4 code behind `gemma4` feature flag  
**Scale/Scope**: Single-user local inference; model sizes from ~3.5 GB (E2B) to ~16 GB (26B)

## Constitution Check

*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*

| Principle | Status | Notes |
|-----------|--------|-------|
| I. Library-First | **Pass** | All changes are in the `local-llm` library crate. No new crates needed — Gemma 4 support fits within the existing crate boundary. |
| II. Test-Driven Development | **Pass** | Unit tests for parser, preset config, model-family detection. Live tests (ignored) for real inference. Tests written before implementation. |
| III. Efficiency & Performance | **Pass** | Stateful parser uses string matching (not regex) on the streaming hot path. No unnecessary allocations — parser buffers only during partial delimiter matches. |
| IV. Leverage the Ecosystem | **Pass** | Uses `mistralrs` 0.8 (upstream inference engine) and `hf-hub` (model download). No hand-rolled inference or download logic. |
| V. Provider Agnosticism | **Pass** | `LocalStreamFn` continues to implement `StreamFn`. Gemma 4 support is internal to the local-llm crate — core sees the same `AssistantMessageEvent` stream regardless of model family. |
| VI. Safety & Correctness | **Pass** | `#[forbid(unsafe_code)]` maintained. NaN detection produces an error event, not a panic. Parser state machine has no unreachable states. |
| Crate count | **Pass** | No new crates. Changes are within `swink-agent-local-llm` (crate 5 of 11). |
| MSRV | **Pass** | 1.88, edition 2024. No MSRV change needed. |

**Post-Phase-1 re-check**: All gates still pass. The `MultimodalModelBuilder` branching adds model-family-specific logic but remains within the existing `ChatBackend` implementation — no new abstraction layers or crates.

## Project Structure

### Documentation (this feature)

```text
specs/041-adapter-gemma4-local/
├── plan.md              # This file
├── spec.md              # Feature specification
├── research.md          # Phase 0 research findings
├── data-model.md        # Entity definitions and state machines
├── quickstart.md        # Usage examples and build instructions
├── checklists/
│   └── requirements.md  # Spec quality checklist
└── tasks.md             # Phase 2 output (generated by /speckit.tasks)
```

### Source Code (repository root)

```text
local-llm/
├── Cargo.toml                 # mistralrs version bump, gemma4 feature flag
├── src/
│   ├── lib.rs                 # Updated re-exports (cfg-gated)
│   ├── model.rs               # ModelConfig::is_gemma4(), MultimodalModelBuilder branch in ChatBackend::build()
│   ├── preset.rs              # Gemma4_E2B/E4B/26B variants, DEFAULT_LOCAL_PRESET_ID conditional
│   ├── convert.rs             # <|think|> injection in LocalConverter::system_message()
│   ├── stream.rs              # ChannelThoughtParser, parser dispatch based on model family
│   ├── loader.rs              # No changes (LoaderBackend trait unchanged)
│   ├── embedding.rs           # No changes
│   ├── error.rs               # No changes
│   └── progress.rs            # No changes
└── tests/
    └── local_live.rs          # New ignored live tests for Gemma 4 inference
```

**Structure Decision**: All changes are within the existing `local-llm/` crate. No new files needed — `ChannelThoughtParser` lives in `stream.rs` alongside the existing `extract_thinking_delta()`. The `ToolCallParser` (P3) will also live in `stream.rs` when implemented.

## Design Decisions

### D1: Builder Branching in ChatBackend::build()

Branch on `ModelConfig::is_gemma4()` within the existing `ChatBackend::build()` method. Both `GgufModelBuilder` and `MultimodalModelBuilder` return `mistralrs::Model`, so the `LoaderBackend` trait, `LazyLoader`, and all downstream code remain unchanged.

```
ChatBackend::build(config)
  ├── config.is_gemma4() == true  → MultimodalModelBuilder::new(repo_id, vec![filename]).build()
  └── config.is_gemma4() == false → GgufModelBuilder::new(repo_id, vec![filename]).build()
```

### D2: Parser Dispatch

The `StreamState::process_content_delta()` method currently calls `extract_thinking_delta()` unconditionally. For Gemma 4, it dispatches to `ChannelThoughtParser` instead. The parser selection is determined once at `StreamState` construction time based on model family (not per-chunk).

The `ChannelThoughtParser` is a field on `StreamState` (behind `#[cfg(feature = "gemma4")]`). When the feature is disabled, the code path compiles away entirely.

### D3: Thinking Token Injection

`<|think|>` is prepended to the system prompt in `LocalConverter::system_message()` when:
1. The model is Gemma 4 (`ModelConfig::is_gemma4()`)
2. Thinking is enabled (determined via `ModelSpec` capabilities or explicit config)

This requires threading model config information into the converter. The current `convert_context_messages(context)` signature gains a `config: &ModelConfig` parameter.

### D4: SmolLM3-3B Remains Default, Gemma 4 Opt-In

`DEFAULT_LOCAL_PRESET_ID = "smollm3_3b"` unconditionally. Gemma 4 E2B (~5 GB safetensors, needs ~10 GB RAM) is too large to be a safe default on all hardware. Developers with capable machines (16+ GB RAM) opt in via `ModelPreset::Gemma4E2B` or environment variables. No Ollama dependency for any default path.

### D5: Tool Call Parser (P3, deferred)

The `ToolCallParser` will follow the same stateful pattern as `ChannelThoughtParser` — a struct on `StreamState` that processes each chunk and emits tool call events. It parses `<|tool_call>call:{name}{args}<tool_call|>` format. Implementation deferred until core text + thinking is stable.

## Complexity Tracking

No constitution violations to justify. All changes fit within the existing crate boundary, use the existing `LoaderBackend` trait, and add no new abstraction layers.
