# R@5 Strategy — synthesis after the failure-mode audit

**Date:** 2026-04-17  
**Inputs:** `docs/audit-r5-failure-modes.md`, Reranker V2 Phase 1 calibration (PR #1031), Phase 2 in flight (Stack v2 corpus build)  
**Goal:** decide which levers to fund (and in what order) to move v3-test R@5 from the current ~63% (permissive) toward 80% (sharp goal) and 90% (stretch).

## Where we actually are

The audit measured two R@5 numbers on v3 test:

| Match mode | R@1 | R@5 | R@20 |
|---|---|---|---|
| Strict `(origin, name, line_start)` | 33.0% | 52.3% | 70.6% |
| Permissive (basename + name fallback) | 41.3% | 63.3% | 81.7% |

The permissive number matches the documented v1.27.0 baseline (R@5 64.2%) within 0.9pp. The 11pp strict↔permissive gap is **eval-data drift** — gold paths and line numbers in `v3_test.json` have moved as the corpus evolved (chunk renames, line-shift on edit, worktree-derived gold). Treating strict as the ground truth understates real retrieval quality by ~11pp.

**So the real distance to targets is:**

- 63% → 80% sharp goal: **+17pp**
- 63% → 90% stretch: **+27pp**

## Failure-mode breakdown (20 near-misses, rank 6-20)

| Mode | N | % near-misses | R@5 ceiling if fully solved |
|---|---|---|---|
| `near_dup_crowding` | 12 | 60% | +11.0pp |
| `wrong_abstraction_top_too_big` | 5 | 25% | +4.6pp |
| `wrong_abstraction_top_too_small` | 4 | 20% | +3.7pp |
| `truncated_gold` | 3 | 15% | +2.8pp |
| `unexplained` | 3 | 15% | +2.8pp |
| `eval_artifact_*` | 2 | 10% | +1.8pp |

Modes overlap (a single query can be both `near_dup_crowding` and `wrong_abstraction_*`), so additive ceilings are **not** the expected lift. Realistic lift per lever is ~30% of its ceiling — a fully working MMR will likely capture 3-5pp of the 11pp ceiling, not all of it.

## Levers, in priority order

### Tier 1 — ship before Reranker V2 lands

Independent of Phase 2/3 outcome. Each ships standalone, each has a clean v3-dev sanity-check gate.

#### 1. Eval-data hygiene (no R@5 change, fixes baseline reading)

**Problem:** 12 of 109 queries match by `name`-only (chunk renamed/moved); 2 queries have gold in `.claude/worktrees/`. Both are eval artifacts, not retrieval failures. Without fixing them, we can't tell if a lever moved real R@5 or just shifted strict↔permissive.

**Lever:** regenerate `v3_test.json` against the current production index. Re-judge any gold whose `(origin, name, line_start)` no longer resolves. Fold name-only matches into acceptable answers (or update gold to point at the moved chunk).

**Expected lift on real R@5:** 0pp. **Expected lift on strict R@5:** +10-11pp (just by removing the artifact). This is a **measurement** lever — pre-requisite for trusting the rest.

**Effort:** ~half a day script + a Gemma re-judge pass on the 14 affected queries.

#### 2. MMR re-rank on top-K pool (`near_dup_crowding`, 12 queries)

**Problem:** 60% of near-misses have ≥3 chunks in top-5 from the same file or with the same function name. Example: a `multi_step` query gets 5 results, 4 from `src/search/router.rs`. Diversity-starved.

**Lever:** apply Maximum Marginal Relevance with λ ≈ 0.5-0.7 on the top-K=20 pool before truncating to K=5. Penalize candidates that are too similar (cosine) to already-selected ones. Optional: weight the diversity penalty by `(file_match + name_match)` so same-file/same-name penalties are stronger than pure embedding similarity.

**Expected lift:** +3-5pp realistic, +11pp ceiling.

**Effort:** 1-2 days. No model change. Wire into `search_filtered`'s post-rank step. Sanity-check on v3 dev (R@5 should not regress on `identifier_lookup` queries — those want the most-similar match).

#### 3. Chunk-type aware boost (`wrong_abstraction_*`, 9 queries)

**Problem:** 45% of near-misses (combined directions) have a length mismatch between the query's intended granularity and what's in top-5. Type_filtered query for "struct definitions in src/cli/commands/infra" gets `Cli` (huge orchestrator struct) at rank 1 instead of small `BatchInput`.

**Lever:** read intent signal from query (e.g., "struct" → struct type, "function" → function type, length adjectives like "small/large"). When intent is detectable, boost the matching `chunk_type` and demote the others. When undetectable, no-op.

**Expected lift:** +2-3pp realistic, +8pp ceiling.

**Effort:** 2-3 days. Lightweight rule-based or a small classifier. Wire as a final-stage scoring tweak. Sanity-check that conceptual_search doesn't regress (those queries want abstraction-mixing).

### Tier 2 — gated on Phase 2 reranker corpus quality

#### 4. Reranker V2 (Phase 2 in flight, Phase 3 next)

**Problem:** `unexplained` queries (15% of near-misses) — embedding-space lexical mismatch. The dense embedder has the gold near top-20 but can't quite get it to top-5 because the surface vocabulary doesn't align.

**Lever:** cross-encoder trained on (query, positive, negative) triples. Phase 2 builds the 200k corpus from Stack v2 (9 languages, function-level chunks with docstrings, BGE-large hard negatives). Phase 3 trains. Wire as a final re-rank pass on the top-50 pool.

**Expected lift:** +5-10pp realistic on R@5. Catches `unexplained` directly and likely also improves `wrong_abstraction_*` (cross-encoders learn relevance signals independent of chunk length).

**Effort:** Phase 2 in flight (~32h GPU wall). Phase 3 ~12-24h training + inference wiring. Total ~3-5 days from now to deployment-ready.

**Risk:** Reranker V2 pilot was −5.5pp R@1. The 4 prereqs (200k+ pairs, code-pretrained base, RRF fusion, top-K input) must all be satisfied. Phase 1 calibration cleared #1 (200k Gemma labels at 98.3% Claude agreement). Phase 2 needs to clear #2 by using a code-aware base. If we screw any of these up, this lever moves backwards.

### Tier 3 — only if Tier 1 + Tier 2 don't reach 80%

#### 5. Chunker fix for `truncated_gold` (3 queries, ceiling +2.8pp)

**Problem:** gold chunks <5 lines have weak embedding signal. Tree-sitter currently extracts the bare struct/signature without surrounding doc context.

**Lever:** when a chunk would be <5 lines, prepend the leading docstring/comment block (or the function header line if no doc). Re-embed. Schema-level — requires reindex.

**Expected lift:** +1-2pp realistic.

**Effort:** ~3 days (chunker change + reindex + re-embed). Schema migration if we surface "expanded" length as a distinct field. Sanity-check that chunk count doesn't blow up (we add to short chunks, not all chunks).

#### 6. Per-category HyDE (parked)

**Problem:** v2-era HyDE data showed +14pp on structural queries, −22pp on conceptual. The router-style "wins-vanish" pattern struck again — cross-category, the average was net-negative.

**Lever:** generate hypothetical-document expansions only for the categories where v2 data showed a positive lift; bypass for the rest. Requires the calibrated classifier we don't have yet.

**Expected lift:** speculative. v2 showed +5-15pp on the right categories.

**Effort:** untested on v3. Re-validate with current router + Gemma 4 31B.

#### 7. ColBERT-class late interaction (parked, architectural)

**Problem:** dense single-vector retrieval has a fundamental ceiling. Cross-encoders catch some of this; ColBERT's per-token interaction catches more.

**Lever:** add a `Reranker` trait, implement ColBERT as a 2-stage re-ranker first (no per-token index). If it wins, build the full per-token index.

**Expected lift:** +10-25pp R@5 on benchmarks (paper numbers; not our domain).

**Effort:** 1-3 months architectural lift. Not the right next step — only after exhausting Tier 1 and 2.

## Decision matrix

| Lever | Ceiling | Realistic | Effort | Order | Why this rank |
|---|---|---|---|---|---|
| Eval-data hygiene | — | — | 0.5d | **1** | Without it, we can't measure anything else |
| MMR re-rank | +11pp | +3-5pp | 1-2d | **2** | Biggest mode, no model change, fast win |
| Chunk-type boost | +8pp | +2-3pp | 2-3d | **3** | Second-biggest mode, low risk |
| Reranker V2 (Phase 2/3) | — | +5-10pp | ~5d | **4** | Already in flight; gated on corpus quality |
| Chunker fix (truncated) | +3pp | +1-2pp | 3d | **5** | Reindex cost; smaller lift |
| Per-category HyDE | speculative | ? | 1w | **6** | Re-validate before committing |
| ColBERT | +20pp paper | unknown for code | 1-3mo | **7** | Last-resort architectural lift |

**Stack to hit the sharp goal (80% R@5):**

Eval-hygiene (baseline-reading fix) → MMR (+3-5pp) → Reranker V2 (+5-10pp) → chunk-type boost (+2-3pp). Sum: **~10-18pp** realistic, baseline 63% → **73-81%**. Hits 80% if the upper end of each lever materializes.

P(hitting 80% with this stack) ≈ 60%. Below that, add chunker fix + per-category HyDE.

**Stack to hit the stretch goal (90% R@5):**

Above + ColBERT-class late interaction OR substantial corpus growth. Below 50% probability even with everything funded.

## What's *not* on the list

- **Centroid classifier polish**. Pilot was −4.6pp; even a calibrated classifier likely net-negative for the wrong-alpha cost asymmetry. Better to remove the router entirely once Reranker V2 absorbs that responsibility.
- **Code-aware embedder swap**. v9-200k was −13.8pp R@1; CodeBERT and CodeT5+ are untested but the v9-200k miss suggests "fine-tuned on code" isn't enough — needs to be code-pretrained, and that's a Reranker V2 base-model decision, not an embedder swap.
- **Knowledge-augmented retrieval (call/type graph filter)**. Multi-step queries are weakest; the graph could help. Parked because untested on v3 and the surface area is large (graph-weighted scoring requires query-side intent).

## Wins-vanish sanity check

Every prior lever-shipping cycle hit the same pattern: per-category gain in dev → muted gain (or regression) on test through the router. Evidence: centroid classifier −4.6pp, reranker pilot −5.5pp, full v3 alpha sweep where only `cross_language` transferred. The only structural defense is **end-to-end v3-dev validation before merging each Tier-1 lever** — never just per-mode improvement, always end-to-end R@K shift on dev.
