# SymForge v4.0.0 Baseline Evaluation

Date: 2026-03-31

Server under test:
- Binary: `C:\Users\rakovnik\.symforge\bin\symforge.exe`
- Reported version: `symforge 4.0.0`
- Protocol observed: newline-delimited JSON-RPC over stdio

Repository under test:
- Path: `E:\project\symforge`
- Commit: `bdddc47`

## Method

1. Start the installed `v4.0.0` server directly over stdio.
2. Initialize MCP and enumerate tools, resources, and prompts.
3. Run `index_folder` against the repository before the test matrix.
4. Exercise every advertised tool at least once.
5. Use `dry_run` for write-capable tools where supported.
6. Record success/failure behavior, notable output quality, and interface issues.

## Capability Inventory

Advertised surface:
- Tools: 25
- Resources: 4
- Prompts: 3

Tools:
- `analyze_file_impact`
- `batch_edit`
- `batch_insert`
- `batch_rename`
- `delete_symbol`
- `diff_symbols`
- `edit_within_symbol`
- `explore`
- `find_dependents`
- `find_references`
- `get_file_content`
- `get_file_context`
- `get_repo_map`
- `get_symbol`
- `get_symbol_context`
- `health`
- `index_folder`
- `insert_symbol`
- `inspect_match`
- `replace_symbol_body`
- `search_files`
- `search_symbols`
- `search_text`
- `validate_file_syntax`
- `what_changed`

Resources:
- `symforge://repo/health`
- `symforge://repo/outline`
- `symforge://repo/map`
- `symforge://repo/changes/uncommitted`

Prompts:
- `symforge-architecture`
- `symforge-review`
- `symforge-triage`

## Core Findings

### What Worked Well

- `index_folder` reindexed the repo reliably and quickly. Baseline run: `Indexed 521 files, 13434 symbols.`
- Read/search/navigation tools were consistently fast and useful:
  - `get_repo_map`
  - `search_symbols`
  - `search_text`
  - `search_files`
  - `get_file_context`
  - `get_file_content`
  - `get_symbol`
  - `get_symbol_context`
  - `find_references`
  - `find_dependents`
  - `inspect_match`
  - `explore`
- Repo-state tools worked:
  - `health`
  - `what_changed`
  - `diff_symbols`
  - `validate_file_syntax`
  - `analyze_file_impact`
- Single-symbol edit tools behaved well in `dry_run`:
  - `replace_symbol_body`
  - `edit_within_symbol`
  - `insert_symbol`
  - `delete_symbol`
- `batch_rename` worked in `dry_run` and returned a sensible preview of rename impact.
- Resources and prompts were available and usable. They are not empty stubs.

### Important Limitations

- The server is daemon-backed and preserves prior state across sessions. Initial `health` before reindex reflected an older shared index, not a clean isolated session. For fair comparisons, explicit `index_folder` is mandatory at the start of every run.
- `get_symbol` on a missing symbol returned a helpful textual fallback but did not set `isError=true`. This is user-friendly, but it means clients cannot rely on error status alone to distinguish success from "not found".
- Some outputs contain decorative box-drawing characters that showed mojibake when consumed directly from PowerShell JSON-RPC. That may be a Windows terminal encoding issue, but it still affects raw harness readability.

### Batch Tool Ergonomics

- `batch_insert` worked with structured object targets:
  - Example shape: `targets: [{ path, name }, ...]`
- `batch_edit` worked with structured typed operations:
  - Example shape: `operation: { type: "edit_within", old_text, new_text }`
  - Example shape: `operation: { type: "delete" }`

But:
- `batch_insert` failed with shorthand string targets such as `execution/test_release_ops.py::test_preflight_steps_include_version_sync`
- `batch_edit` failed with shorthand string edit forms such as `path::name => delete`
- Those failures surfaced as generic `-32602` deserialization errors

Assessment:
- The batch tools are functionally present.
- Their object payloads work.
- Their shorthand payload story is brittle enough that I would consider it a usability issue in `v4.0.0`.

## Tool-by-Tool Baseline

| Tool | Result | Notes |
| --- | --- | --- |
| `health` | Pass | Useful status, file counts, watcher status, git temporal summary. Shared daemon state means it is not isolated by default. |
| `index_folder` | Pass | Reliable baseline reset. Required for fair comparison. |
| `get_repo_map` | Pass | Fast, compact, high-signal project overview. |
| `search_symbols` | Pass | Precise and fast on known symbol lookup. |
| `search_text` | Pass | Worked on known string queries and returned contextual symbol hits. |
| `search_files` | Pass | Good ranked path discovery. |
| `get_file_context` | Pass | Strong file-level outline and context summary. |
| `get_file_content` | Pass | Exact line-range reads worked cleanly. |
| `get_symbol` | Pass with caveat | Good symbol body retrieval. Missing symbol returns advice text instead of an MCP error. |
| `get_symbol_context` | Pass | Useful usage summary; compact mode is valuable. |
| `find_references` | Pass | Clear caller/reference listing. |
| `find_dependents` | Pass | Helpful file-level dependency summary. |
| `inspect_match` | Pass | Good local symbol context around a line hit. |
| `what_changed` | Pass | Cleanly reported no local changes. |
| `diff_symbols` | Pass | Useful symbol-level change summary across refs. |
| `explore` | Pass | Good concept-oriented discovery. |
| `validate_file_syntax` | Pass | Correctly validated workflow YAML. |
| `analyze_file_impact` | Pass | After clean reindex, correctly reported `indexed and unchanged`. |
| `replace_symbol_body` | Pass | `dry_run` preview worked. |
| `edit_within_symbol` | Pass | `dry_run` preview worked. |
| `insert_symbol` | Pass | `dry_run` preview worked. |
| `delete_symbol` | Pass | `dry_run` preview worked. |
| `batch_rename` | Pass | `dry_run` preview was useful and credible. |
| `batch_insert` | Pass with caveat | Structured object payload worked; shorthand string target failed with deserialization error. |
| `batch_edit` | Pass with caveat | Structured typed operation payload worked; shorthand string form failed with deserialization error. |

## Resources and Prompts

Resources:
- All 4 advertised resources were readable.
- `repo/health`, `repo/outline`, and `repo/map` were genuinely useful.
- `repo/changes/uncommitted` matched `what_changed`.

Prompts:
- All 3 advertised prompts returned structured prompt payloads.
- They appear useful as planning surfaces rather than placeholders.

## Overall Opinion

`v4.0.0` is already a serious and useful MCP surface.

Strengths:
- Strong search/read/navigation primitives
- Fast enough to use constantly
- Good `dry_run` support on single-symbol editing tools
- Resources and prompts add real value
- `index_folder` provides a clean reset primitive for evaluation

Weaknesses:
- Shared daemon state means behavior is not isolated unless the client deliberately reindexes
- Batch edit interfaces are more brittle than the rest of the surface
- Error signaling is inconsistent in a few places
- Direct raw protocol consumption on Windows showed some encoding ugliness in decorated text output

Bottom line:
- I would trust `v4.0.0` for read-heavy and symbol-targeted work.
- I would be more cautious with batch editing until payload ergonomics and error clarity improve.
- For comparison against the newer server, the common-surface test should start with explicit `index_folder` and then rerun this same matrix. New-only tools should be evaluated separately rather than mixed into the baseline score.

## Reuse Guidance For The Next Comparison

When the newer server is installed:
1. Test the same repository path.
2. Test the same commit if possible.
3. Start with `index_folder`.
4. Run the same 25 shared-tool cases first.
5. Evaluate any new tools in a separate section.
6. Compare:
   - tool inventory
   - startup/index behavior
   - output quality
   - error quality
   - batch editing ergonomics
   - resources/prompts availability
