# SymForge v4.9.9 Evaluation

Date: 2026-03-31

Server under test:
- Binary: `C:\Users\rakovnik\.symforge\bin\symforge.exe`
- Reported version: `symforge 4.9.9`
- Protocol observed: newline-delimited JSON-RPC over stdio

Repository under test:
- Path: `E:\project\symforge`
- Commit: `bdddc47`

## Method

1. Start the installed `v4.9.9` server directly over stdio.
2. Initialize MCP and enumerate tools, resources, and prompts.
3. Run `index_folder` against the repository before the matrix.
4. Re-run the shared 25-tool baseline cases from the `v4.0.0` evaluation.
5. Exercise all 5 newly advertised tools.
6. Read all resources and fetch all prompts.
7. Include a few negative cases to judge error quality, not only happy paths.

## Capability Inventory

Advertised surface:
- Tools: 30
- Resources: 4
- Prompts: 6

Tools newly present relative to the `v4.0.0` baseline:
- `ask`
- `context_inventory`
- `conventions`
- `edit_plan`
- `investigation_suggest`

Prompts newly present relative to the `v4.0.0` baseline:
- `symforge-debug`
- `symforge-onboard`
- `symforge-refactor`

## Important Context For Comparison

The repository now contains the earlier baseline report:
- `docs/evaluations/2026-03-31-symforge-v4.0.0-baseline.md`

That means:
- `index_folder` reported `522 files, 13445 symbols`
- `what_changed(uncommitted=true)` surfaced that report file as the only uncommitted change

So file and symbol counts are not directly comparable to the `v4.0.0` run unless that report file is excluded. The functional comparison is still valid.

## Core Results

### Shared 25-Tool Surface

The shared surface remained strong.

Read/search/navigation tools:
- `health`
- `get_repo_map`
- `search_symbols`
- `search_text`
- `search_files`
- `get_file_context`
- `get_file_content`
- `get_symbol`
- `get_symbol_context`
- `find_references`
- `find_dependents`
- `inspect_match`
- `explore`

All of the above worked cleanly and remained fast.

Repo-state tools:
- `index_folder` worked and remained the required reset step for fair evaluation
- `validate_file_syntax` worked
- `analyze_file_impact` correctly returned `indexed and unchanged`
- `diff_symbols` worked
- `what_changed` worked, correctly reporting the uncommitted baseline report file

Edit tools:
- `replace_symbol_body` `dry_run` worked
- `edit_within_symbol` `dry_run` worked
- `insert_symbol` `dry_run` worked
- `delete_symbol` `dry_run` worked
- `batch_rename` `dry_run` worked
- `batch_insert` with structured object payload worked
- `batch_edit` with structured typed operations worked

Negative and edge behavior:
- `get_symbol` on a missing symbol still returns a helpful advisory text response rather than an MCP error state
- `batch_insert` shorthand string form still fails with `-32602`
- `batch_edit` shorthand string form still fails with `-32602`

## New Tool Evaluation

### `ask`

Result: good

Observed behavior:
- For `where is release_metadata_is_aligned defined`, it routed to `search_symbols`
- The route explanation was explicit: `[Routed to: search_symbols(query=\"release_metadata_is_aligned\")]`

Assessment:
- Useful convenience layer
- Good for natural-language entry
- Routing transparency is valuable

### `conventions`

Result: strong

Observed behavior:
- Returned project-wide style heuristics:
  - error handling mix
  - naming conventions
  - test counts
  - file organization
  - complexity notes
  - common imports

Assessment:
- High-value orientation tool
- Better than ad hoc style inference from scattered reads
- One of the best additions in `v4.9.9`

### `context_inventory`

Result: useful

Observed behavior:
- Reported session context inventory after prior tool calls
- Counted loaded symbols/files and estimated token usage

Assessment:
- Practical and measurable
- Helps the agent reason about loaded context rather than guessing
- This is a meaningful addition for context hygiene

### `edit_plan`

Result: useful but lightweight

Observed behavior:
- For `execution/release_ops.py`, it suggested a short sequence:
  - `get_file_context`
  - `get_symbol`
  - an edit tool
  - `analyze_file_impact`

Assessment:
- Good as a small workflow recommender
- Not deep, but not pretending to be
- Useful as a guardrail for disciplined edits

### `investigation_suggest`

Result: mixed

Observed behavior:
- Suggested symbols referenced but not yet loaded
- In this run it suggested things like `run` and `str`

Assessment:
- The idea is good
- Current output can be noisy or too literal
- This tool feels less mature than `ask`, `conventions`, `context_inventory`, and `edit_plan`

## Resources and Prompts

Resources:
- All 4 resources were readable and consistent with the tool surface
- `repo-health`, `repo-outline`, `repo-map`, and `repo-changes-uncommitted` all worked

Prompts:
- All 6 prompts returned structured workflows, not placeholders
- New prompts appear more operational and instructional than the `v4.0.0` set

New prompt observations:
- `symforge-debug` is especially relevant and more actionable than a generic prompt
- `symforge-onboard` is useful for layered repository orientation
- `symforge-refactor` provides a concrete planning scaffold

Assessment:
- Prompt surface is materially better than the `v4.0.0` baseline

## Direct Impressions vs v4.0.0

### Improvements

- The surface is broader and more coherent
- New tools are mostly useful, especially:
  - `ask`
  - `conventions`
  - `context_inventory`
  - `edit_plan`
- Prompt support is meaningfully stronger
- Shared core performance remains strong
- `analyze_file_impact` behaved more cleanly in this run than the one ambiguous early baseline reading during the old evaluation

### Things That Did Not Improve Enough

- Batch shorthand payload ergonomics are still broken
- Error reporting for those deserialization failures is still generic
- Missing-symbol lookups still prefer advisory text over explicit machine-level error semantics
- `investigation_suggest` still feels undercooked relative to the rest of the surface

## Overall Opinion

`v4.9.9` is a real improvement over `v4.0.0`.

The important point is not just the extra tool count. The product feels more complete:
- stronger prompt layer
- better context-awareness
- a useful natural-language router
- explicit project convention discovery
- better support for disciplined workflows around edits and investigations

My ranking of the new additions:
1. `conventions`
2. `ask`
3. `context_inventory`
4. `edit_plan`
5. `investigation_suggest`

Bottom line:
- I would prefer working with `v4.9.9` over `v4.0.0`
- The read/search/orientation experience is clearly better
- The prompt/tool ecosystem feels more intentional
- The biggest remaining weakness is still interface polish on batch operations and a few semantics around error reporting

If I reduce it to one sentence:
- `v4.9.9` feels like `v4.0.0` plus a genuinely useful agent-facing guidance layer, but it still carries some rough edges in the most advanced editing interfaces.
