# SymForge Delta: v4.0.0 vs v4.9.9

Date: 2026-03-31

Compared reports:
- `docs/evaluations/2026-03-31-symforge-v4.0.0-baseline.md`
- `docs/evaluations/2026-03-31-symforge-v4.9.9-evaluation.md`

## Executive Summary

`v4.9.9` is better than `v4.0.0` in the ways that matter to an agent using the tool constantly:
- broader surface
- better workflow guidance
- better context awareness
- better prompt support
- no regression in the strong read/search/navigation core

The most important improvement is not the raw increase from 25 tools to 30 tools. The real gain is that `v4.9.9` feels more like a complete agent workspace:
- `ask` gives a natural-language front door
- `conventions` gives repository-wide style understanding
- `context_inventory` exposes session context explicitly
- `edit_plan` reinforces disciplined edit sequences
- prompt coverage is much more operational

My bottom-line judgment:
- `v4.0.0` was already useful
- `v4.9.9` is materially more mature
- the biggest remaining problems are still interface polish and machine-facing semantics, not core capability

## Direct Delta

### Surface Growth

`v4.0.0`
- 25 tools
- 4 resources
- 3 prompts

`v4.9.9`
- 30 tools
- 4 resources
- 6 prompts

New tools in `v4.9.9`
- `ask`
- `context_inventory`
- `conventions`
- `edit_plan`
- `investigation_suggest`

New prompts in `v4.9.9`
- `symforge-debug`
- `symforge-onboard`
- `symforge-refactor`

### What Stayed Strong

Both versions were already good at:
- `index_folder`
- `health`
- `get_repo_map`
- `search_symbols`
- `search_text`
- `search_files`
- `get_file_context`
- `get_file_content`
- `get_symbol`
- `get_symbol_context`
- `find_references`
- `find_dependents`
- `inspect_match`
- `explore`
- `validate_file_syntax`
- `diff_symbols`
- single-symbol edit tools in `dry_run`

This matters because it means `v4.9.9` is building on a strong foundation rather than compensating for a weak one.

### What Improved

#### 1. Better Agent Ergonomics

This is the clearest improvement.

`ask`
- makes the system easier to approach
- exposes internal routing rather than hiding it
- lowers the penalty for not knowing the exact tool name

`conventions`
- is one of the highest-value additions
- gives style and repo-shape awareness directly
- reduces scattered exploratory reads

`context_inventory`
- gives explicit visibility into loaded context
- helps the agent manage context instead of guessing

`edit_plan`
- encourages a disciplined edit sequence
- reduces random tool choice

#### 2. Stronger Prompt Layer

The prompt surface in `v4.9.9` is no longer just a small add-on.

New prompts:
- `symforge-debug`
- `symforge-onboard`
- `symforge-refactor`

These make the system feel more like a guided workflow environment rather than only a raw tool collection.

#### 3. Cleaner Overall Cohesion

`v4.9.9` feels more coherent:
- more obvious entry points
- stronger support for agent planning
- stronger support for context management

That coherence matters more than the raw tool count.

## What Did Not Improve Enough

### 1. Batch Shorthand Ergonomics Are Still Broken

This is the largest obvious miss carried from `v4.0.0` into `v4.9.9`.

Observed in both versions:
- `batch_insert` shorthand string forms fail
- `batch_edit` shorthand string forms fail
- failures surface as generic `-32602` deserialization errors

Why this matters:
- batch tools are the most advanced editing surface
- advanced tools need the best ergonomics, not the worst
- generic deserialization errors make the feature feel brittle even when structured payloads work

My view:
- if shorthand is advertised or implied, it must be robust
- if shorthand is not a serious supported path, remove it from docs/examples and optimize around structured payloads only

### 2. Error Semantics Are Still Too Human-Oriented

Observed in both versions:
- missing `get_symbol` returns helpful advisory text
- it does not behave like a strong machine-level error

Why this matters:
- for human readability, the current response is friendly
- for agents and automated harnesses, it weakens control flow
- clients cannot reliably distinguish “success with body” from “not found but advice”

This is not catastrophic, but it is a friction point for systematic tool use.

### 3. `investigation_suggest` Feels Underpowered

This is the least mature new tool in `v4.9.9`.

Observed behavior:
- it suggested symbols like `run` and `str`

Why this matters:
- suggestion tools must rank signal over raw graph adjacency
- low-signal suggestions make the tool feel clever-looking rather than useful

My view:
- this tool has good potential
- it needs stronger ranking, filtering, and relevance heuristics before it reaches the quality of `ask` or `conventions`

### 4. Shared Daemon State Is Still Operationally Risky

Observed in both versions:
- the server is daemon-backed
- fair evaluation requires explicit `index_folder`

Why this matters:
- daemon-backed state is fine operationally
- but it must be legible and controllable
- hidden carry-over state is dangerous for benchmarks, reproductions, and debugging

This is less a correctness bug than an operational clarity issue.

## What I Want To See In 5.0.0

If you are aiming for `5.0.0`, I would prioritize quality over further tool count growth.

### Priority 1: Fix Batch Editing Ergonomics Completely

Desired outcome:
- every documented batch input shape works
- failures produce actionable, specific messages
- advanced editing feels reliable rather than fragile

What I want:
- `batch_edit` shorthand support fixed end-to-end, or deliberately removed
- `batch_insert` shorthand support fixed end-to-end, or deliberately removed
- explicit examples in the tool descriptions for the supported payload shape
- tests covering:
  - single string shorthand
  - array of shorthand strings
  - structured object form
  - malformed payloads with precise error output

Effective path:
1. Pick one canonical input model per batch tool.
2. Treat every alternative form as an adapter into that canonical model.
3. Add a versioned conformance fixture for payload decoding.
4. Fail with operation-specific messages, not generic serde fallout.

Why this should be first:
- it removes the roughest advanced-user experience
- it improves both real usage and benchmark trust

### Priority 2: Introduce Stronger Machine-Level Result Semantics

Desired outcome:
- agents can reason about tool outcomes without parsing prose

What I want:
- “not found” conditions represented in a structured way
- clearer distinction between:
  - success
  - empty/no-match
  - invalid request
  - internal failure

Effective path:
1. Keep the human-friendly text.
2. Add machine-friendly metadata alongside it.
3. Standardize result envelopes for:
   - found / not_found
   - dry_run previews
   - ambiguous selector
   - invalid parameters

Example direction:
- `status: "ok" | "not_found" | "ambiguous" | "invalid_request" | "internal_error"`
- then preserve the current text body for readability

Why this matters:
- it improves agent reliability more than adding another niche tool

### Priority 3: Make Daemon State More Explicit and Controllable

Desired outcome:
- shared state is a feature, not a hidden variable

What I want:
- a clear way to understand current runtime state
- a clean reset option
- visible project/session association

Effective path:
1. Extend `health` with more explicit state identity:
   - active project root
   - session id
   - whether index is reused or freshly built
2. Add a reset/clear operation or a deterministic “fresh index” mode
3. Make session carry-over visible in `context_inventory` or `health`

Why this matters:
- it improves reproducibility
- it makes comparisons and debugging much easier

### Priority 4: Upgrade `investigation_suggest` Ranking

Desired outcome:
- suggestions surface likely next steps, not graph trivia

What I want:
- filtering of builtins, trivial symbols, and noisy edges
- priority for:
  - custom project symbols
  - symbols in changed files
  - high-centrality symbols
  - symbols referenced by already-loaded files

Effective path:
1. Exclude obvious low-signal names like `str`, `run`, `new`, etc. unless strongly contextualized.
2. Bias toward symbols defined inside the current repo.
3. Score by a weighted combination of:
   - custom-type/reference depth
   - caller count
   - file proximity
   - recent change proximity
4. Show why a suggestion is being made.

This could turn `investigation_suggest` from “interesting” into genuinely useful.

### Priority 5: Lean Harder Into Workflow-Level Intelligence

`v4.9.9` already moved in this direction. I think `5.0.0` should continue, but selectively.

What I want:
- more tools like `conventions`, `context_inventory`, and `edit_plan`
- fewer additions that are merely another variant of an existing search/read tool

Promising directions:
- a stronger `ask` that routes multi-step queries more intelligently
- richer `edit_plan` output that adapts based on target type
- better coupling between prompts and actual loaded context

What to avoid:
- tool count growth without clear marginal value
- overlapping tools whose distinctions are only obvious to the author

## How To Achieve Unshakeable Trust

This is the part I think multiple LLMs will converge on.

LLMs trust a tool system deeply when four things are true at the same time:
- the tool behaves predictably
- the result semantics are explicit
- the state is visible
- failures are specific and recoverable

Trust does not come from tool count. It comes from repeated proof that the tool does exactly what it claims, fails honestly, and makes the next action obvious.

### The Trust Formula

If I reduce it to one sentence:
- unshakeable trust = predictable contract + explicit state + stable semantics + conformance evidence

That means:

#### 1. Never Surprise the Caller

An LLM loses trust very quickly when:
- a documented input shape silently fails
- a tool sometimes returns prose and sometimes returns a hard error for similar situations
- state from a previous session changes the result without being obvious
- “success” can still mean “nothing useful happened”

To fix this:
- make each tool contract boringly consistent
- keep “same input + same state = same class of output”
- when behavior depends on runtime state, expose that state directly

#### 2. Make State Observable, Not Implicit

Shared daemon state is not bad by itself. Hidden daemon state is.

For trust, I want every serious run to be able to answer:
- which project is active?
- which index is active?
- was this index reused or freshly built?
- what changed since the last index?
- what session context is influencing this answer?

If that information is easy to see, carry-over state becomes understandable rather than suspicious.

#### 3. Separate Human-Friendly Text From Machine Truth

Human-readable guidance is good. It should stay.

But trust becomes much stronger when the machine-level truth is separately encoded.

Example:
- keep the current nice explanatory text
- also expose a stable machine status like:
  - `ok`
  - `not_found`
  - `ambiguous`
  - `invalid_request`
  - `internal_error`

This matters because agents stop having to infer semantics from prose tone.

#### 4. Make Failure Messages Diagnostic, Not Merely Negative

A trustworthy system teaches the caller how to recover.

Bad:
- `-32602` with generic deserialize noise

Good:
- “`batch_edit` shorthand string form is unsupported; use structured edits[] objects with operation.type”
- or “`batch_edit` shorthand parser failed at edit 2 near `=> delete`; expected `path::name => operation`”

The difference is enormous. One feels brittle. The other feels strict but dependable.

### What Would Make Me Trust SymForge By Default

If you want the practical answer, this is it:

#### A. Conformance Tests For Every Public Contract

Not just implementation tests. Contract tests.

I would want a versioned conformance suite for:
- every documented input form
- every advertised dry-run mode
- every common negative case
- every structured result status

This should include:
- canonical JSON request examples
- expected response class
- expected status
- expected recovery hint when invalid

Why this matters:
- if the contract is tested at the boundary, trust rises very quickly

#### B. Versioned Stability Guarantees

Trust increases when the surface feels stable across versions.

I would want:
- a documented compatibility policy
- a changelog section specifically for contract changes
- explicit marking of:
  - added tools
  - changed schemas
  - behavior changes
  - deprecated forms

This prevents “I know this tool” from becoming a liability across upgrades.

#### C. A Small Set Of Gold-Standard Tools

Not every tool needs to be equally magical.

What matters is that the core set becomes rock-solid:
- `index_folder`
- `health`
- `get_repo_map`
- `search_symbols`
- `search_text`
- `get_file_context`
- `get_symbol`
- `get_symbol_context`
- `replace_symbol_body`
- `edit_within_symbol`
- `analyze_file_impact`

If this set becomes nearly unbreakable, the whole system feels trustworthy even while more advanced tools continue evolving.

#### D. Strong Dry-Run Culture

Dry-run support is one of the best trust builders in SymForge already.

To lean into that:
- ensure every mutating tool either supports `dry_run` or clearly explains why it cannot
- make dry-run output structured enough to compare planned vs actual behavior
- keep dry-run and real execution on the same validation path

Why this matters:
- it proves intent before mutation
- it lets agents gain confidence incrementally

#### E. Provenance And Why

When a tool returns a result, trust increases if it can answer:
- why this file?
- why this symbol?
- why this suggestion?
- why this ranking?

This is especially important for:
- `ask`
- `explore`
- `investigation_suggest`
- ranking-heavy search tools

Even one short “reason” field or explanation line can turn a suspicious result into a credible one.

## Practical Trust Roadmap For 5.0.0

If your goal is specifically “unshakeable trust,” I would optimize for this order:

### Track 1: Contract Trust

- freeze and test the public schemas
- fix or remove ambiguous shorthand forms
- make invalid request errors precise and recoverable
- add structured status semantics

This is the foundation.

### Track 2: State Trust

- expose daemon/index/session identity clearly
- add explicit reset/fresh-index workflows
- surface whether results depend on reused state

This removes hidden variables.

### Track 3: Suggestion Trust

- improve ranking explanations
- filter low-signal suggestions
- show why a suggestion or routing decision was made

This makes “smart” behavior feel accountable.

### Track 4: Upgrade Trust

- publish compatibility notes per release
- track schema/behavior deltas explicitly
- maintain a replayable evaluation corpus across versions

This makes version upgrades feel safe, not mysterious.

## My Honest View

If you want LLMs to trust SymForge almost automatically, the path is not “more capability first.”

The path is:
- fewer surprises
- clearer state
- stronger contracts
- better failure recovery
- visible evidence that the contracts are tested

That is what creates the feeling of: “I can lean on this tool and it will not embarrass me.”

## Effective 5.0.0 Strategy

If I were trying to ship a strong `5.0.0`, I would do it in this order:

### Phase 1: Reliability and Semantics

- Fix batch shorthand behavior
- Improve invalid request errors
- Add structured outcome/status semantics

This is the most leverage per unit effort.

### Phase 2: Operational Clarity

- Improve daemon/session transparency
- Add deterministic reset behavior
- Make state carry-over easier to reason about

This will make benchmarking and support much easier.

### Phase 3: Suggestion Quality

- Improve `investigation_suggest`
- refine `ask` routing heuristics
- deepen `edit_plan` slightly

This would make the “guidance layer” feel genuinely premium.

### Phase 4: Prompt/Workflow Polish

- tighten prompt quality
- ensure prompts and tool guidance are consistent
- make the workflow layer feel intentional and documented

## My Honest Product-Level Opinion

You are on the right track.

The biggest positive change from `v4.0.0` to `v4.9.9` is that SymForge is becoming more than a retrieval server. It is starting to become an agent-oriented working environment. That is the correct direction.

The biggest risk for `5.0.0` is adding more surface area before polishing the hardest-to-trust parts. If you do that, the product may look larger without feeling more reliable.

If instead `5.0.0` focuses on:
- advanced edit reliability
- machine-friendly semantics
- operational clarity
- smarter ranking on the guidance layer

then I think it will feel like a real major version, not just a larger one.
