# MentisDB 0.8.6 Implementation Plan

## Goal
Close ~38% of remaining LoCoMo lexical gaps and push LongMemEval R@5 above 70%.

## Three Parallel Workstreams

### Workstream A: Irregular Verb Lemma Expansion
**Agent:** `lexicographer`  
**Files:** `src/search/lexical.rs`, `src/search/stemmer.rs` (new)  
**Depends on:** nothing  
**Blocks:** benchmarks

**Design:**
1. Add `IRREGULAR_VERBS: phf::Map<&str, &str>` constant — ~100 entries mapping English irregular past tense → base form (went→go, gave→give, ran→run, saw→see, etc.)
2. In `normalize_lexical_tokens()`, after Porter stemming, check each token against the irregular verb map. If the raw (pre-stem) token matches a key, also emit the base form as an additional token.
3. This is query-time only — no index rebuild needed. The index already contains stemmed forms; the query just gets extra matching power.
4. Add a `normalize_lexical_tokens_with_lemmas()` variant for query paths that includes lemma expansion, keeping the index path unchanged.

**Test plan:**
- Unit test: `normalize_lexical_tokens_with_lemmas("went")` produces both "went"-stem and "go"-stem tokens
- Integration: LoCoMo benchmark with `--skip-vectors` before/after

**Estimated size:** ~150 lines new code + ~100 lines test data

---

### Workstream B: Lightweight Reranking via Reciprocal Rank Fusion
**Agent:** `reranker`  
**Files:** `src/search/ranked.rs` (new), `src/lib.rs` (query_ranked integration)  
**Depends on:** nothing  
**Blocks:** benchmarks

**Design:**
1. On `RankedSearchQuery`, add `enable_reranking: bool` (default false) and `rerank_k: usize` (default 50).
2. When reranking is enabled and `query_ranked()` returns top-K, run a second scoring pass:
   - Produce a lexical-only ranking (set vector=0) over top `rerank_k` candidates
   - Produce a vector-only ranking (set lexical=0) over top `rerank_k` candidates  
   - Merge via Reciprocal Rank Fusion: `score = 1/(k + rank_lexical) + 1/(k + rank_vector)` where k=60 (standard RRF constant)
3. The reranked list replaces the original top-K.
4. This is pure arithmetic — no LLM, no model, no external service.

**Test plan:**
- Unit test: given two rankings with different order, RRF produces merged ranking
- Unit test: `enable_reranking = true` on `RankedSearchQuery` triggers rerank path
- Integration: LoCoMo benchmark before/after

**Estimated size:** ~200 lines new code

---

### Workstream C: Per-Field BM25 DF Cutoffs
**Agent:** `bm25-tuner`  
**Files:** `src/search/lexical.rs`, `src/search/bm25.rs` (if exists, else lexical.rs)  
**Depends on:** nothing  
**Blocks:** benchmarks

**Design:**
1. Replace the single `BM25_DF_CUTOFF: f32 = 0.3` constant with a per-field struct:
   ```rust
   struct Bm25DfCutoffs {
       content: f32,  // default 0.3
       tags: f32,     // default 0.5 (tags are high-frequency by nature)
       concepts: f32, // default 0.4
   }
   ```
2. In the BM25 scoring loop, when computing IDF for a term, select the cutoff based on which field(s) the term appeared in during indexing.
3. Terms that appear in tags but not content should use the tags cutoff (more permissive, since tag vocabularies are smaller).
4. Add `Bm25DfCutoffs` to `MentisDb` as a configurable field with builder method.

**Test plan:**
- Unit test: term at 35% DF in content gets filtered (cutoff 0.3), same term at 35% DF in tags passes (cutoff 0.5)
- Integration: LoCoMo benchmark before/after

**Estimated size:** ~100 lines modified code

---

## Integration & Benchmark Phase
**Agent:** `benchmarker`  
**Depends on:** A, B, C all complete  
**Files:** benchmark runners, `changelog.txt`, version bump

1. Run LoCoMo 2-persona, 10-persona, and LongMemEval benchmarks with all three features enabled
2. Compare against 0.8.5 baselines (LoCoMo 10p: 74.6% R@10, LME: 67.6% R@5)
3. Tune parameters (verb list size, RRF k, per-field cutoffs) for best R@10
4. Update `changelog.txt` with 0.8.6 section
5. Bump version to 0.8.6
6. Blog post for 0.8.6

## Parallelism Diagram

```
   A: Irregular Verbs  ──┐
                          ├──►  Benchmark & Release
   B: RRF Reranking   ──┤
                          │
   C: Per-Field DF     ──┘
```

All three workstreams are independent — no shared files, no merge conflicts.
