# SymForge Live Evaluation — Release Improvement Report

**Date:** 2026-06-16  
**Reporter:** Cursor agent session (SymForge MCP live battery)  
**SymForge version:** `7.27.0` (`Cargo.toml`)  
**Runtime:** `local_process` / `SYMFORGE_NO_DAEMON` equivalent (in-process index, no daemon proxy)  
**Mandate:** Read-only audit + dry-run edits only. No product-repo source edits.

**Related artifacts**

| Artifact | Relationship |
|----------|----------------|
| [`docs/archived/SYMFORGE_MCP_BUG_REPORT_2026-06-03.md`](../archived/SYMFORGE_MCP_BUG_REPORT_2026-06-03.md) | Prior cross-repo audit (7.18.0) |
| [`docs/archived/SYMFORGE_MCP_BUG_REPORT_2026-06-03_VERDICTS.md`](../archived/SYMFORGE_MCP_BUG_REPORT_2026-06-03_VERDICTS.md) | Root-cause verification for June issues |
| [`docs/research/81-index-recall-evidence-index.md`](./81-index-recall-evidence-index.md) | Active index-recall program (T2.x) — overlaps SF-LIVE-002/003 |
| [`docs/v8-gap-closure-plan.md`](../v8-gap-closure-plan.md) | Binding v8 rework gates (H1–H8) |
| `C:\AI_STUFF\PROGRAMMING\remort.md` | External shallow eval (Mistral Vibe, 2026-06-15) — contrast only |

---

## Executive summary

SymForge **7.27.0** remains a strong **local-first code intelligence layer** for agent workflows. The highest-value surface is unchanged:

- `health_compact` / `health`
- `index_folder`
- `get_file_context` / `get_repo_map`
- `get_symbol` (path-scoped)
- `search_symbols` + `search_text` on Rust-heavy repos
- `conventions` (now language-aware)
- dry-run structural edits

On a **large Rust monorepo** (`Agent_Army_Professionals`, 1,396 files / 36,613 symbols, ~1.1s index), symbol lookup, caller analysis, and file compression are production-usable.

On a **TypeScript / NestJS / Angular repo** (`testpilot`, 413 files / 11,287 symbols), outline and convention tools work well, but **index recall and reference completeness regress** — the same class of gap the **81 index-recall** program is meant to close.

### Top release priorities (ordered)

| Priority | ID | Theme | Target release |
|----------|-----|-------|----------------|
| P0 | SF-LIVE-002 | TypeScript method symbols missing from `search_symbols` | 7.28.x patch + 8.1 recall gate |
| P0 | SF-LIVE-003 | `find_references` incomplete on cross-file / frontend TS | 7.28.x + index-recall T2 |
| P1 | SF-LIVE-001 | `find_dependents` file graph is crate-coarse and misleading | 7.28.x UX contract |
| P1 | SF-LIVE-004 | `ask` compound queries stop at first symbol hop | 7.28.x routing |
| P2 | SF-LIVE-005 | Trust headers not loud enough on heuristic reference results | 7.28.x / STEL metadata |
| P2 | SF-LIVE-006 | External evaluators over-rate tools (no stress path) | Docs + `sf-bench` rows |
| Carry | SF-LIVE-007 | Single-project `index_folder` rebind (by design) | v8 product docs / future multi-root |

### Contrast with external shallow eval (`remort.md`)

Mistral's report rated nearly every tested tool ⭐⭐⭐⭐⭐ and evaluated **~9 of 32 tools** on small Rust repos. This battery confirms the **directional praise** for read/outline tools but refutes **blanket 5-star reference trust**, especially on web stacks. Treat `remort.md` as onboarding material, not a regression spec.

---

## Mini-spec

**Objective:** Produce a durable, actionable improvement report from live SymForge MCP usage on real repos, with enough reproduction detail for maintainers to file issues, write tests, and gate v8 / index-recall work.

**Non-goals**

- Do not modify SymForge product source in this document task.
- Do not modify audited product repos.
- Do not re-litigate June audit items already fixed in 7.18.x–7.27.0 except where live behavior regressed or remains partially open.

**Contracts under test**

- Tools advertising exact / constrained reference behavior must not silently omit known call sites.
- `search_symbols` must find symbols that `get_symbol` can retrieve by path.
- File-level dependency tools must state granularity (file import vs crate/module vs symbol).
- Parse and framework limitations must remain explicit in output (7.27 behavior here is **good** — preserve in v8).

**Acceptance for this report**

- Environment + repo stats recorded.
- Per-issue repro, expected, actual, impact, suggested fix, and test recipe.
- Findings mapped to existing programs (index-recall, v8 gates) where applicable.

---

## Environment

| Field | Value |
|-------|-------|
| OS | Windows 10 (win32 10.0.26200) |
| SymForge | `7.27.0` via MCP `user-symforge` |
| Mode | `local_process`, fresh index per `index_folder` |
| Session | `local-process-23140` |

### Repos exercised

| Repo | Role | Files | Symbols | Index time |
|------|------|------:|--------:|-----------|
| `Agent_Army_Professionals` | Large Rust monorepo (AAP) | 1,396 | 36,613 | ~1,120 ms |
| `testpilot` | NestJS + Angular web app | 413 | 11,287 | ~219 ms |
| `terminal-commander` | Rust workspace (smoke) | 524 | 11,837 | fast |
| `symforge` | Self-dogfood (smoke) | 366 | 16,302 | fast |

---

## Test matrix (tool × repo)

| Tool | AAP | testpilot | Notes |
|------|:---:|:---------:|-------|
| `health_compact` | PASS | PASS | Parse quarantine, git temporal, token savings |
| `index_folder` | PASS | PASS | Rebinds project; prior index discarded |
| `search_symbols` | PASS | **FAIL** | Misses `startExploration` entirely on testpilot |
| `get_symbol` | PASS | PASS | Path-scoped retrieval works when name search fails |
| `get_symbol_context` | PASS | PASS | SF-002 fix verified (no self-caller on delegation) |
| `find_references` | PASS | **PARTIAL** | Rust exact; TS misses frontend + repo-wide holes |
| `find_dependents` | **PARTIAL** | — | 510-file crate-coarse graph; suspicious leaf files |
| `get_file_context` | PASS | PASS | Best tool family; framework notes on `app.html` |
| `get_repo_map` | PASS | PASS | |
| `explore` | PASS | PASS | Keyword/concept expansion — not semantic search |
| `search_text` | PASS | PASS | Regex `Subscription\[\]` hits correct |
| `conventions` | PASS | PASS | Rust vs TS detection correct |
| `ask` | PASS | PARTIAL | Tool catalog good; compound Q incomplete |
| `validate_file_syntax` | — | PASS | `import('rxjs').Subscription[]` → ok + limitation note |
| `what_changed` | PASS | — | Noisy on dirty AAP worktree (scratch files) |
| `investigation_suggest` | — | PASS | Session-aware next steps |
| `replace_symbol_body` dry-run | PASS | — | Clear safety headers |
| `analyze_file_impact` | — | PASS | |
| `diff_symbols` | PASS | — | Empty diff correct for clean HEAD |
| Structural edit suite (7 tools) | NOT RUN | NOT RUN | Recommend sf-bench / harness rows |
| `checkpoint_now` | NOT RUN | NOT RUN | Prior audit fixed daemon path; re-verify in proxy mode |
| `find_dependents` + `get_symbol_context` cross-check | PASS | PARTIAL | See issues |

**June fixes confirmed still green on 7.27.0**

- SF-002: controller `startExploration` does not list itself as caller; delegation surfaced as unresolved same-name.
- SF-003: `workflow-builder.component.ts` validates `ok` with parser-limitation note.
- SF-004: `app.html` reports `ok` with Angular framework-limitation note (improved vs raw `partial`).
- SF-010: `ask("what tools for impact?")` returns impact-analysis catalog.
- SF-011: `conventions` on testpilot reports TypeScript / NestJS — not Rust-biased.

---

## Issue index

| ID | Severity | Category | Title | Target |
|----|----------|----------|-------|--------|
| SF-LIVE-001 | High | UX / correctness | `find_dependents` file graph is crate-coarse; lists unrelated files | 7.28.x |
| SF-LIVE-002 | High | Index recall | `search_symbols` returns zero hits for known TS methods | 7.28.x + T2 |
| SF-LIVE-003 | High | Index recall | `find_references` omits known TS call sites (incl. frontend) | 7.28.x + T2 |
| SF-LIVE-004 | Medium | Routing | `ask` does not complete compound definition + import questions | 7.28.x |
| SF-LIVE-005 | Medium | Trust surface | Heuristic reference modes need louder uncertainty | 7.28.x |
| SF-LIVE-006 | Low | Measurement | External evals over-score without cross-repo stress | Docs / sf-bench |
| SF-LIVE-007 | Low | Product | Single active project per session (document, don't "fix" in 7.x) | v8 docs |

---

## SF-LIVE-001 — `find_dependents` file graph is crate-coarse

**Severity:** High (trust)  
**Status:** Confirmed on 7.27.0

**Tool:** `find_dependents`  
**Repo:** `Agent_Army_Professionals`

**Repro**

```text
index_folder(path="C:\AI_STUFF\PROGRAMMING\Agent_Army_Professionals")
find_dependents(path="crates/aap-db/src/stores/work_item.rs", compact=true, limit=12)
```

**Actual**

- Reports **510 files** depend on `work_item.rs`.
- Lists `crates/aap-agents/src/actor_runtime/actor.rs` with `(7 refs: 7 call)`.

**Ground truth (shell)**

```text
rg "work_item|WorkItemStore|aap_db" crates/aap-agents/src/actor_runtime/actor.rs
→ No matches
```

`actor.rs` imports only actor-runtime modules (`kameo`, `super::error`, etc.) — no `aap-db` path.

**Expected**

Either:

1. Only files with **direct module/import edges** to `work_item.rs` appear, or
2. Output explicitly states **crate-level reverse dependency** and does not imply per-file call/import evidence.

**Impact**

Agents treating `find_dependents` as authoritative blast-radius analysis will over-scope refactors on large monorepos. This is the same *trust* failure class as the historical SF-001 audit, but at **file granularity**.

**Suggested fix**

- Split modes: `granularity=file_import | crate | module` (default `file_import`).
- When returning crate-coarse results, suppress per-file `N refs: N call` tallies unless backed by edges.
- Add regression fixture: file with zero textual/import link to target must not appear in `file_import` mode.

**Test recipe**

- Extend `tests/find_dependents_pass2.rs` with AAP-shaped fixture: dependent crate member file with no import edge → must not list target file as dependent in strict mode.

---

## SF-LIVE-002 — `search_symbols` misses known TypeScript methods

**Severity:** High (recall)  
**Status:** Confirmed on 7.27.0  
**Program overlap:** 81 index-recall / A-029 T2

**Tool:** `search_symbols`  
**Repo:** `testpilot`

**Repro**

```text
index_folder(path="C:\AI_STUFF\PROGRAMMING\testpilot")
search_symbols(query="startExploration", limit=10)
```

**Actual**

```text
No symbols matching 'startExploration'.
```

**Ground truth**

Method exists in at least:

- `backend/src/modules/testing/testing.controller.ts:55`
- `backend/src/modules/testing/services/testing.service.ts:669`
- `frontend/src/app/core/services/api.service.ts:426`

`get_symbol(name="startExploration", path="backend/.../testing.service.ts")` **succeeds**.

**Expected**

`search_symbols` returns all indexed method symbols matching the name (or states index gap with `result_status`).

**Impact**

Agents follow tool guidance ("find by name first") and conclude the symbol does not exist, wasting turns on `search_text` / manual grep.

**Suggested fix**

- Audit TS symbol emission for class methods (decorated controllers, services, Angular services).
- Align `search_symbols` index with `get_symbol` path resolution.
- Add testpilot-derived golden: `startExploration` → ≥3 hits across backend + frontend.

**Test recipe**

- Fixture repo with Nest controller + service + frontend API method same name → `search_symbols` must return ≥3.

---

## SF-LIVE-003 — `find_references` incomplete on TypeScript

**Severity:** High (recall)  
**Status:** Confirmed on 7.27.0  
**Program overlap:** 81 index-recall / A-029 T2 (primary)

**Tool:** `find_references`, `get_symbol_context`  
**Repo:** `testpilot`

**Repro**

```text
find_references(name="startExploration", compact=true, limit=25)
```

**Actual (repo-wide)**

3 references in 2 files:

- `testing.controller.ts:60`
- `testing.service.runs.spec.ts:769,778`

**Missing (shell ground truth)**

- Definition site `testing.service.ts:669` (visible via `get_symbol` but not in repo-wide reference list)
- Frontend `api.service.ts:426`

**Scoped call (partial success)**

```text
get_symbol_context(name="startExploration", path="backend/.../testing.service.ts", sections=["callers"])
```

→ 3 callers including controller — **backend only**.

**Expected**

Repo-wide `find_references` lists all call sites across backend + frontend, or returns `constrained` / `partial` with explicit omitted-count reason and next-step (`search_text`, path-scoped pass).

**Impact**

Cross-stack refactors (API rename, endpoint tracing) fail silently. Directly hurts v8 **H6** equivalence on web corpora.

**Suggested fix**

- Close T2 source-class gaps for TS member calls and cross-package references.
- Ensure spec files and frontend services share reference extraction with backend methods.
- Surface `Completeness: partial (frontend package not in reference index)` when applicable.

**Test recipe**

- Golden route: `find_references(startExploration)` → includes `api.service.ts` line ≥1.
- Battery row in `sf-bench` under `trace` / `find_references` intent for testpilot SHA.

---

## SF-LIVE-004 — `ask` stops on first hop of compound questions

**Severity:** Medium  
**Status:** Confirmed on 7.27.0 (improved vs 7.18; still incomplete)

**Tool:** `ask`  
**Repo:** `testpilot`

**Repro**

```text
ask(query="Where is TestingController defined and what module imports it?")
```

**Actual**

- Routes to `search_symbols(query="TestingController")` with `inferred` confidence.
- Suggested chain printed, but **module import question unanswered**.
- Text fallback mentions `testing.module.ts` without executing `find_references` / `get_file_content`.

**Expected**

Multi-clause questions decompose into a **plan** (define → imports) or execute chained tools automatically under STEL L2+.

**Suggested fix (7.x)**

- Extend `ask` planner: detect `and what module imports` → auto-chain `find_references` or `get_file_context` imports section.
- Return `result_status=partial` when only first clause satisfied.

**Suggested fix (v8)**

- STEL controller issues multi-step plan with ledger; compact surface returns plan + first result + pending steps.

**Test recipe**

- Golden ask route from [`docs/research/A-028-golden-routes.md`](./A-028-golden-routes.md) pattern: compound controller lookup → module file cited.

---

## SF-LIVE-005 — Trust headers for heuristic reference modes

**Severity:** Medium  
**Status:** Enhancement (7.27 partially addresses)

**Observation**

7.27 already emits strong trust lines (`Trust: constrained`, `Parse state`, `Completeness`). Still, repo-wide `find_references` on `startExploration` reports `parsed | full` while omitting known sites — **`full` is false** from an agent's perspective.

**Suggested fix**

- Distinguish **index completeness** vs **query completeness**.
- When reference extraction skipped a language/package boundary, emit `Completeness: partial (cross-package references omitted)`.

**v8 alignment**

- Map to `symforge/result_status` and STEL judge inputs so H6 failures are measurable.

---

## SF-LIVE-006 — External evaluators need a stress corpus

**Severity:** Low (process)  
**Status:** Recommendation

Shallow agent evals (e.g. Mistral `remort.md`) test happy-path Rust reads and produce inflated star ratings. SymForge maintainers should publish a **minimal public stress checklist** (≤30 min):

1. AAP `find_dependents` on `work_item.rs` + shell cross-check one alleged dependent.
2. testpilot `search_symbols(startExploration)` + `find_references` vs `rg`.
3. testpilot `validate_file_syntax` on `workflow-builder.component.ts`.
4. `ask` compound controller question.
5. `replace_symbol_body` dry-run on real symbol.

Pin as `docs/research/live-eval-checklist.md` or sf-bench appendix.

---

## SF-LIVE-007 — Single-project session (documented constraint)

**Severity:** Low (product)  
**Status:** By design on 7.27.0

Each `index_folder` rebinds `repo_root` and replaces the in-memory index. Cross-repo analysis requires re-indexing or multiple sessions.

**Recommendation for v8 rework**

- Do not promise multi-root in compact STEL until durable project switching + snapshot identity exists.
- Operator docs: "one active project per MCP session" with explicit `index_folder` cost.
- Future: project registry under `.symforge/` with named slots (out of 8.0 scope unless already scheduled).

---

## Strengths to preserve through v8 rework

These behaviors **must not regress** during STEL / compact-surface work:

| Behavior | Evidence |
|----------|----------|
| Fast indexing at 1k+ files | AAP 1,396 files / ~1.1s |
| `get_file_context` compression + trust headers | AAP `work_item.rs` ~6.5k vs whole-file tokens |
| Language-aware `conventions` | Rust on AAP, TypeScript on testpilot |
| Parser honesty | `validate_file_syntax` limitation notes (SF-003) |
| Framework partial classification | Angular `app.html` (SF-004) |
| SF-002 caller hygiene | `startExploration` controller context |
| Tool catalog via `ask` | Impact-analysis group listing |
| Dry-run edit safety | `replace_symbol_body` on `claim_one` |
| `health_compact` identity fields | `project_id`, `index_generation`, parse quarantine |

---

## Release mapping (recommended)

### 7.28.x (patch — trust + recall)

| Item | Work |
|------|------|
| SF-LIVE-001 | `find_dependents` granularity + honest labeling |
| SF-LIVE-002 | TS method symbol indexing |
| SF-LIVE-003 | Reference extraction gaps (coordinate with index-recall T2) |
| SF-LIVE-004 | `ask` multi-clause chaining |
| SF-LIVE-005 | Query-level completeness metadata |

### 8.0.0 (STEL — do not hide recall debt)

- Compact `symforge` intent routes must **BYPASS** or **degrade** when recall confidence low (per H3/H6).
- Carry trust metadata into compact responses (postcard + `_meta`).

### 8.1.0 (index-recall program exit)

- Gate on testpilot + AAP golden routes for `search_symbols` / `find_references`.
- sf-bench rows: `trace`, `impact-analysis`, `explore` on pinned SHAs.

### AAP embed path (committed)

- Prioritize SF-LIVE-001/003 fixes — AAP is the committed tight-integration repo ([`docs/v8-aap-integration.md`](../v8-aap-integration.md)).

---

## Suggested regression harness additions

```text
tests/fixtures/live_eval/
  testpilot_start_exploration_search_symbols.json   # expect >=3 symbol hits
  testpilot_start_exploration_find_refs.json        # expect api.service.ts hit
  aap_work_item_find_dependents_strict.json         # actor.rs must NOT appear
  ask_compound_controller_module.json               # expect module path in output
```

Integrate with:

- `cargo test --test live_eval_regression` (new integration binary), or
- sf-bench MCP replay rows referencing fixture hashes.

---

## Verification commands (maintainers)

```bash
# SymForge repo gates (after implementing fixes)
cargo fmt --check
cargo clippy --all-targets -- -- -D warnings
cargo test --all-targets -- --test-threads=1

# Manual replay (Windows paths)
# 1. Index testpilot via MCP index_folder
# 2. search_symbols(query="startExploration") → must not be empty
# 3. find_references(name="startExploration") → must include frontend api.service.ts
# 4. Index AAP → find_dependents strict on work_item.rs → actor.rs absent
```

---

## Appendix A — Mistral `remort.md` score adjustment

| Mistral rating | Adjusted (this battery) | Reason |
|----------------|-------------------------|--------|
| `find_references` ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ Rust / ⭐⭐⭐ web | TS omissions |
| `search_symbols` ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ Rust / ⭐⭐⭐ web | TS method gap |
| `explore` ⭐⭐⭐⭐ | ⭐⭐⭐⭐ (hold) | Not semantic search — document |
| `find_dependents` (not rated) | ⭐⭐⭐ | Crate-coarse graph |
| `ask` (not rated) | ⭐⭐⭐⭐ catalog / ⭐⭐⭐ compound | |

---

## Appendix B — Raw session metrics

| Metric | Value |
|--------|-------|
| AAP parse issues | 4 unexpected partial, 0 failed |
| testpilot parse issues | 0 unexpected; 1 expected framework + 1 expected language partial |
| testpilot token savings (session) | ~30,419 tokens (health counter) |
| AAP git temporal | 500 commits / 90d in 2,088 ms |
| testpilot git temporal | 479 commits / 90d in 919 ms |

---

*End of report. File bugs against SF-LIVE-* IDs. Update [`81-index-recall-evidence-index.md`](./81-index-recall-evidence-index.md) when T2 work closes SF-LIVE-002/003.*
