# MCP Compatibility Specification

This document details the exact changes required for Substrate to be fully compatible with the Model Context Protocol (MCP) revision **2025-03-26** (with notes on 2024-11-05 compatibility).

## Design Principle: Unbuffered Streaming via Streamable HTTP

The 2025-03-26 MCP spec introduces **Streamable HTTP transport** which allows SSE streaming responses. This means we can implement MCP **without buffering** by streaming progress notifications alongside the final result.

```
┌─────────────────────────────────────────────────────────────────┐
│                    Streamable HTTP Flow                          │
│                                                                  │
│  POST /mcp                                                       │
│  { "method": "tools/call", "params": { "name": "claudecode.chat" } }
│       │                                                          │
│       ▼                                                          │
│  HTTP/1.1 200 OK                                                 │
│  Content-Type: text/event-stream                                 │
│       │                                                          │
│       ├──► event: message                                        │
│       │    data: {"method":"notifications/progress",             │
│       │           "params":{"message":"Token: Hello"}}           │
│       │                                                          │
│       ├──► event: message                                        │
│       │    data: {"method":"notifications/progress",             │
│       │           "params":{"message":"Token: world"}}           │
│       │                                                          │
│       └──► event: message                                        │
│            data: {"id":1,"result":{"content":[...]}}  ◄── final  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

### Two Modes of Operation

| Mode | Transport | Streaming | Use Case |
|------|-----------|-----------|----------|
| **Unbuffered** | Streamable HTTP (SSE) | Progress notifications + final result | Real-time UI, long operations |
| **Buffered** | stdio / simple HTTP | Collect stream → single response | Simple clients, legacy |

With Streamable HTTP, Plexus stream events map directly to MCP progress notifications - **no buffering required**.

---

## Fallback: Buffered Mode

For clients that don't support Streamable HTTP (or stdio transport), we fall back to buffering. This section describes that architecture.

```
┌─────────────────────────────────────────────────────────────────┐
│                         Substrate                                │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │                 Transport (stdio / HTTP)                    │  │
│  └──────────────────────────┬─────────────────────────────────┘  │
│                             │                                    │
│               ┌─────────────┴─────────────┐                      │
│               │     Method Router         │                      │
│               │                           │                      │
│               │  "initialize" ──────────┐ │                      │
│               │  "tools/list" ──────────┤ │                      │
│               │  "tools/call" ──────────┼─┼─► MCP Interface      │
│               │                         │ │   (buffering)        │
│               │  "claudecode.chat" ─────┼─┼─► Plexus Interface   │
│               │  "bash.execute" ────────┘ │   (streaming)        │
│               │                           │                      │
│               └─────────────┬─────────────┘                      │
│                             │                                    │
│  ┌──────────────────────────┴─────────────────────────────────┐  │
│  │                                                             │  │
│  │  ┌─────────────────────┐       ┌─────────────────────┐     │  │
│  │  │   MCP Interface     │       │  Plexus Interface   │     │  │
│  │  │   (buffering)       │       │  (streaming)        │     │  │
│  │  │                     │       │                     │     │  │
│  │  │  • Collects stream  │       │  • Real-time events │     │  │
│  │  │  • Returns single   │       │  • Subscriptions    │     │  │
│  │  │    aggregated resp  │       │  • Progressive UI   │     │  │
│  │  │                     │       │                     │     │  │
│  │  └──────────┬──────────┘       └──────────┬──────────┘     │  │
│  │             │                             │                 │  │
│  │             └──────────────┬──────────────┘                 │  │
│  │                            │                                │  │
│  │                            ▼                                │  │
│  │             ┌─────────────────────────────┐                 │  │
│  │             │         Plexus              │                 │  │
│  │             │      (activations)          │                 │  │
│  │             │                             │                 │  │
│  │             │  health │ bash │ claudecode │                 │  │
│  │             │  arbor  │ cone │ ...        │                 │  │
│  │             └─────────────────────────────┘                 │  │
│  │                                                             │  │
│  └─────────────────────────────────────────────────────────────┘  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

### Key Insight: MCP as a View

The MCP interface is a **view** over Plexus, not a replacement:

| Client Type | Interface | Delivery Model | Use Case |
|-------------|-----------|----------------|----------|
| Claude Code | MCP (`tools/call`) | Single buffered response | Tool integration |
| Symbols/Haskell | Plexus (`claudecode.chat`) | Real-time stream events | Progressive UI |
| Web UI | Plexus (subscriptions) | SSE stream | Live updates |

**Same activations, different delivery**:

```rust
// MCP client calls tools/call
// → MCP Interface buffers stream → returns aggregated response

// Native client calls claudecode.chat
// → Plexus Interface streams events → real-time updates
```

### Why Not Replace Streaming?

1. **Progressive UI**: Streaming enables real-time token display
2. **Long-running operations**: `claudecode.chat` can run for minutes
3. **Cancellation**: Streams can be cancelled mid-flight
4. **Resource efficiency**: No need to buffer entire response in memory
5. **Existing clients**: Symbols layer depends on streaming

MCP's single-response model is designed for short-lived tool calls, not extended agent interactions.

---

## Protocol Overview

MCP is a JSON-RPC 2.0 based protocol with three message types:
- **Requests**: Include `id` and `method`, expect response
- **Responses**: Include `id` matching request, contain `result` or `error`
- **Notifications**: Include `method` only (no `id`), no response expected

## Current Substrate vs MCP

| Aspect | Substrate Current | MCP Required |
|--------|-------------------|--------------|
| Message format | JSON-RPC 2.0 | JSON-RPC 2.0 |
| Transport | stdio (newline-delimited) | stdio (newline-delimited) |
| Method naming | `namespace_method` | Flat (`initialize`, `tools/list`) |
| Streaming | Subscription-based | Single response (with optional progress) |
| Handshake | None | `initialize` → `initialized` required |
| State machine | Stateless | Stateful (Uninitialized → Ready) |

## Required Methods

### 1. Lifecycle Methods

#### `initialize` (Request)

**Purpose**: Handshake with version and capability negotiation.

**Request**:
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "initialize",
  "params": {
    "protocolVersion": "2024-11-05",
    "capabilities": {
      "roots": { "listChanged": true },
      "sampling": {}
    },
    "clientInfo": {
      "name": "claude-code",
      "version": "1.0.0"
    }
  }
}
```

**Response**:
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "protocolVersion": "2024-11-05",
    "capabilities": {
      "tools": { "listChanged": true },
      "resources": { "subscribe": true, "listChanged": true },
      "prompts": { "listChanged": true },
      "logging": {}
    },
    "serverInfo": {
      "name": "substrate",
      "version": "0.1.0"
    }
  }
}
```

**Substrate Changes**:
```rust
// Add to plexus router (no namespace prefix)
fn handle_initialize(&self, params: InitializeParams) -> InitializeResult {
    // Validate protocol version
    if !self.supports_version(&params.protocol_version) {
        return Err(JsonRpcError {
            code: -32602,
            message: "Unsupported protocol version",
            data: Some(json!({
                "supported": ["2024-11-05"],
                "requested": params.protocol_version
            }))
        });
    }

    // Build capabilities from registered activations
    let capabilities = ServerCapabilities {
        tools: Some(ToolsCapability { list_changed: true }),
        resources: self.has_arbor().then(|| ResourcesCapability {
            subscribe: true,
            list_changed: true,
        }),
        prompts: None, // Optional
        logging: Some(LoggingCapability {}),
    };

    self.state.set(McpState::Initializing);

    InitializeResult {
        protocol_version: "2024-11-05".into(),
        capabilities,
        server_info: ServerInfo {
            name: "substrate".into(),
            version: env!("CARGO_PKG_VERSION").into(),
        },
    }
}
```

#### `notifications/initialized` (Notification)

**Purpose**: Client acknowledges initialization complete.

**Message**:
```json
{
  "jsonrpc": "2.0",
  "method": "notifications/initialized"
}
```

**Substrate Changes**:
```rust
fn handle_initialized(&self) {
    self.state.set(McpState::Ready);
    // Now accept tools/list, tools/call, etc.
}
```

### 2. Tools Methods

#### `tools/list` (Request)

**Purpose**: Enumerate available tools with JSON Schema definitions.

**Request**:
```json
{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/list",
  "params": {
    "cursor": null
  }
}
```

**Response**:
```json
{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "tools": [
      {
        "name": "claudecode.chat",
        "description": "Chat with a Claude Code session",
        "inputSchema": {
          "type": "object",
          "properties": {
            "session_name": {
              "type": "string",
              "description": "Name of the session"
            },
            "query": {
              "type": "string",
              "description": "The query to send"
            },
            "max_turns": {
              "type": ["integer", "null"],
              "description": "Maximum conversation turns"
            }
          },
          "required": ["session_name", "query"]
        }
      },
      {
        "name": "bash.execute",
        "description": "Execute a bash command",
        "inputSchema": {
          "type": "object",
          "properties": {
            "command": {
              "type": "string",
              "description": "The command to execute"
            }
          },
          "required": ["command"]
        }
      }
    ],
    "nextCursor": null
  }
}
```

**Substrate Changes**:
```rust
fn handle_tools_list(&self, params: ToolsListParams) -> ToolsListResult {
    self.require_state(McpState::Ready)?;

    // Get all activation methods from plexus
    let schema = self.plexus.full_schema();

    // Transform to MCP tool format
    let tools: Vec<McpTool> = schema.activations
        .iter()
        .flat_map(|activation| {
            activation.methods.iter().map(|method| {
                McpTool {
                    // Use dot-notation: "namespace.method"
                    name: format!("{}.{}", activation.namespace, method.name),
                    description: method.description.clone(),
                    input_schema: method.params_schema.clone(),
                }
            })
        })
        .collect();

    // Handle pagination if needed
    let (page, next_cursor) = self.paginate(&tools, params.cursor, 50);

    ToolsListResult {
        tools: page,
        next_cursor,
    }
}
```

#### `tools/call` (Request)

**Purpose**: Invoke a tool and return results.

**Request**:
```json
{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "tools/call",
  "params": {
    "name": "claudecode.chat",
    "arguments": {
      "session_name": "my-session",
      "query": "Hello, world!"
    }
  }
}
```

**Response** (success):
```json
{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Hello! How can I help you today?"
      }
    ],
    "isError": false
  }
}
```

**Response** (tool error):
```json
{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Session 'my-session' not found"
      }
    ],
    "isError": true
  }
}
```

**Substrate Changes**:
```rust
async fn handle_tools_call(&self, params: ToolsCallParams) -> ToolsCallResult {
    self.require_state(McpState::Ready)?;

    // Parse tool name: "namespace.method" → (namespace, method)
    let (namespace, method) = params.name
        .split_once('.')
        .ok_or_else(|| JsonRpcError {
            code: -32602,
            message: format!("Invalid tool name format: {}", params.name),
            data: None,
        })?;

    // Route to plexus activation
    let plexus_method = format!("{}.{}", namespace, method);

    // Collect streaming results into single response
    let mut content = Vec::new();
    let mut is_error = false;

    let stream = self.plexus.call(&plexus_method, params.arguments).await;

    pin_mut!(stream);
    while let Some(event) = stream.next().await {
        match event {
            // Collect text content from stream events
            StreamEvent::Content { text } => {
                content.push(McpContent::Text { text });
            }
            StreamEvent::Error { message } => {
                content.push(McpContent::Text { text: message });
                is_error = true;
            }
            StreamEvent::Complete { result } => {
                // Extract final result if not already captured
                if content.is_empty() {
                    content.push(McpContent::Text {
                        text: serde_json::to_string_pretty(&result)
                            .unwrap_or_default()
                    });
                }
            }
            // Handle other event types...
            _ => {}
        }
    }

    ToolsCallResult { content, is_error }
}
```

### 3. Resources Methods (Optional)

Map Arbor trees/nodes to MCP resources.

#### `resources/list`

```json
{
  "jsonrpc": "2.0",
  "id": 4,
  "method": "resources/list"
}
```

**Response**:
```json
{
  "jsonrpc": "2.0",
  "id": 4,
  "result": {
    "resources": [
      {
        "uri": "arbor://tree/main/node/root",
        "name": "Main Tree Root",
        "mimeType": "application/json"
      }
    ]
  }
}
```

#### `resources/read`

```json
{
  "jsonrpc": "2.0",
  "id": 5,
  "method": "resources/read",
  "params": {
    "uri": "arbor://tree/main/node/abc123"
  }
}
```

**Response**:
```json
{
  "jsonrpc": "2.0",
  "id": 5,
  "result": {
    "contents": [
      {
        "uri": "arbor://tree/main/node/abc123",
        "mimeType": "application/json",
        "text": "{\"role\": \"assistant\", \"content\": \"...\"}"
      }
    ]
  }
}
```

### 4. Prompts Methods (Optional)

Could expose cone sessions as prompt templates.

#### `prompts/list`

```json
{
  "jsonrpc": "2.0",
  "id": 6,
  "method": "prompts/list"
}
```

**Response** (empty if not supported):
```json
{
  "jsonrpc": "2.0",
  "id": 6,
  "result": {
    "prompts": []
  }
}
```

### 5. Utility Methods

#### `ping` (Request)

**Purpose**: Health check / keepalive.

```json
{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "ping"
}
```

**Response**:
```json
{
  "jsonrpc": "2.0",
  "id": 7,
  "result": {}
}
```

#### `notifications/cancelled` (Notification)

**Purpose**: Cancel in-progress request.

```json
{
  "jsonrpc": "2.0",
  "method": "notifications/cancelled",
  "params": {
    "requestId": "3",
    "reason": "User cancelled"
  }
}
```

**Substrate Changes**:
```rust
fn handle_cancelled(&self, params: CancelledParams) {
    // Find the subscription/task by request ID
    if let Some(task) = self.pending_requests.get(&params.request_id) {
        task.cancel();
        self.pending_requests.remove(&params.request_id);
    }
    // Notification - no response sent
}
```

## State Machine

```rust
#[derive(Clone, Copy, PartialEq)]
enum McpState {
    Uninitialized,
    Initializing,
    Ready,
    ShuttingDown,
}

impl Plexus {
    fn require_state(&self, required: McpState) -> Result<(), JsonRpcError> {
        let current = self.mcp_state.get();
        if current != required {
            return Err(JsonRpcError {
                code: -32002,
                message: match current {
                    McpState::Uninitialized => "Server not initialized",
                    McpState::Initializing => "Server initializing",
                    McpState::ShuttingDown => "Server shutting down",
                    _ => "Invalid state",
                }.into(),
                data: None,
            });
        }
        Ok(())
    }
}
```

## Error Codes

| Code | Name | When |
|------|------|------|
| `-32700` | Parse error | Invalid JSON |
| `-32600` | Invalid Request | Not valid JSON-RPC |
| `-32601` | Method not found | Unknown method |
| `-32602` | Invalid params | Wrong params type/value |
| `-32603` | Internal error | Server error |
| `-32002` | Server not initialized | Called before `initialize` |

## Transport Requirements

### stdio (2024-11-05 compatible)

```
┌────────────────────────────────────────────────────────┐
│  stdio Transport                                       │
│                                                        │
│  stdin  ← JSON-RPC requests (newline-delimited)        │
│  stdout → JSON-RPC responses (newline-delimited)       │
│  stderr → Logs (tracing)                               │
│                                                        │
│  COMPLIANT: Messages already newline-delimited         │
│  LIMITATION: No streaming (buffered mode only)         │
└────────────────────────────────────────────────────────┘
```

**Required**: Ensure no embedded newlines in JSON output.

```rust
fn send_message(&self, msg: &Value) {
    let json = serde_json::to_string(msg).unwrap();
    debug_assert!(!json.contains('\n'), "MCP: no embedded newlines");
    println!("{}", json);
}
```

### Streamable HTTP (2025-03-26 - Preferred)

Single endpoint supporting both JSON and SSE responses:

```
┌────────────────────────────────────────────────────────┐
│  Streamable HTTP Transport                             │
│                                                        │
│  POST /mcp                                             │
│    Accept: application/json, text/event-stream         │
│    Body: JSON-RPC request                              │
│                                                        │
│  Response options:                                     │
│                                                        │
│  1. Simple JSON (buffered):                            │
│     Content-Type: application/json                     │
│     { "jsonrpc": "2.0", "id": 1, "result": {...} }    │
│                                                        │
│  2. SSE Stream (unbuffered):                           │
│     Content-Type: text/event-stream                    │
│     event: message                                     │
│     data: {"method":"notifications/progress",...}      │
│     ...                                                │
│     event: message                                     │
│     data: {"id":1,"result":{...}}  ← final response   │
│                                                        │
│  GET /mcp                                              │
│    Server-initiated messages (optional)                │
│                                                        │
│  DELETE /mcp                                           │
│    Session termination                                 │
│                                                        │
│  Headers:                                              │
│    Mcp-Session-Id: <uuid>  (session tracking)          │
│    Last-Event-ID: <id>     (resumability)              │
└────────────────────────────────────────────────────────┘
```

### Transport Selection

```rust
enum McpTransport {
    /// Newline-delimited JSON-RPC over stdio
    /// - Buffered mode only
    /// - 2024-11-05 compatible
    Stdio,

    /// Streamable HTTP with optional SSE
    /// - Unbuffered streaming via progress notifications
    /// - Session management via Mcp-Session-Id header
    /// - Resumability via Last-Event-ID
    /// - 2025-03-26
    StreamableHttp {
        endpoint: String,  // e.g., "/mcp"
    },
}
```

### Backwards Compatibility

For clients that may use either transport:

```rust
async fn detect_transport(req: &Request) -> McpTransport {
    // Check Accept header
    let accept = req.headers().get("Accept").unwrap_or("");

    if accept.contains("text/event-stream") {
        // Client supports streaming
        McpTransport::StreamableHttp { ... }
    } else {
        // Fall back to buffered JSON
        McpTransport::Stdio  // or simple HTTP
    }
}
```

## Content Types

MCP supports three content types in tool responses:

### Text Content
```json
{ "type": "text", "text": "Result text" }
```

### Image Content
```json
{
  "type": "image",
  "data": "base64-encoded-data",
  "mimeType": "image/png"
}
```

### Embedded Resource
```json
{
  "type": "resource",
  "resource": {
    "uri": "arbor://tree/main/node/123",
    "mimeType": "application/json",
    "text": "{...}"
  }
}
```

## Plexus Router Changes

### Method Registration

```rust
impl Plexus {
    pub fn new() -> Self {
        let mut plexus = Self::default();

        // Register MCP protocol methods (no namespace prefix)
        plexus.register_method("initialize", Self::handle_initialize);
        plexus.register_notification("notifications/initialized", Self::handle_initialized);
        plexus.register_method("ping", Self::handle_ping);
        plexus.register_method("tools/list", Self::handle_tools_list);
        plexus.register_method("tools/call", Self::handle_tools_call);
        plexus.register_method("resources/list", Self::handle_resources_list);
        plexus.register_method("resources/read", Self::handle_resources_read);
        plexus.register_method("prompts/list", Self::handle_prompts_list);
        plexus.register_method("prompts/get", Self::handle_prompts_get);
        plexus.register_notification("notifications/cancelled", Self::handle_cancelled);

        plexus
    }
}
```

### Method Resolution

```rust
fn route(&self, method: &str, params: Value) -> Result<Response> {
    // 1. Check MCP protocol methods first (exact match)
    if let Some(handler) = self.mcp_methods.get(method) {
        return handler(self, params);
    }

    // 2. Check namespaced activation methods
    if let Some((namespace, method_name)) = method.split_once('.') {
        if let Some(activation) = self.activations.get(namespace) {
            return activation.call(method_name, params);
        }
    }

    // 3. Legacy underscore format (backward compat)
    if let Some((namespace, method_name)) = method.split_once('_') {
        if let Some(activation) = self.activations.get(namespace) {
            return activation.call(method_name, params);
        }
    }

    Err(JsonRpcError::method_not_found(method))
}
```

## The Buffering Layer

The core of MCP compatibility is a buffering layer that collects Plexus stream events and aggregates them into a single MCP response.

### Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                     MCP Buffering Layer                          │
│                                                                  │
│  tools/call request                                              │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  1. Parse tool name: "claudecode.chat"                   │    │
│  │  2. Extract namespace: "claudecode", method: "chat"      │    │
│  └─────────────────────────────────────────────────────────┘    │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  3. Call Plexus activation (returns Stream<Event>)       │    │
│  └─────────────────────────────────────────────────────────┘    │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  4. Buffer: Collect stream events                        │    │
│  │                                                          │    │
│  │     Stream<Event>                   Buffer               │    │
│  │     ─────────────                   ──────               │    │
│  │     Start { ... }        →          (skip)               │    │
│  │     Content { "Hello" }  →          text += "Hello"      │    │
│  │     Content { " world" } →          text += " world"     │    │
│  │     ToolUse { ... }      →          contents.push(...)   │    │
│  │     Complete { ... }     →          (terminate)          │    │
│  │                                                          │    │
│  └─────────────────────────────────────────────────────────┘    │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  5. Aggregate into MCP response                          │    │
│  │                                                          │    │
│  │     { "content": [{ "type": "text", "text": "..." }],    │    │
│  │       "isError": false }                                 │    │
│  └─────────────────────────────────────────────────────────┘    │
│       │                                                          │
│       ▼                                                          │
│  tools/call response                                             │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

### Implementation

```rust
/// The MCP buffering layer - sits in front of Plexus
pub struct McpInterface {
    plexus: Arc<Plexus>,
    state: Arc<RwLock<McpState>>,
}

impl McpInterface {
    /// Buffer a Plexus stream into an MCP response
    async fn buffer_stream(
        stream: impl Stream<Item = ActivationEvent>,
    ) -> ToolsCallResult {
        let mut contents = Vec::new();
        let mut is_error = false;
        let mut text_buffer = String::new();

        pin_mut!(stream);
        while let Some(event) = stream.next().await {
            match event {
                // Skip lifecycle events
                ActivationEvent::Start { .. } => {}

                // Accumulate text content
                ActivationEvent::Content { text } => {
                    text_buffer.push_str(&text);
                }

                // Capture tool invocations
                ActivationEvent::ToolUse { tool_name, input, .. } => {
                    contents.push(McpContent::Text {
                        text: format!("[Tool: {}] {:?}", tool_name, input)
                    });
                }

                // Capture errors
                ActivationEvent::Error { message } => {
                    is_error = true;
                    contents.push(McpContent::Text { text: message });
                }

                // Stream complete
                ActivationEvent::Complete { .. } => break,
            }
        }

        // Prepend accumulated text
        if !text_buffer.is_empty() {
            contents.insert(0, McpContent::Text { text: text_buffer });
        }

        ToolsCallResult { content: contents, is_error }
    }

    /// Handle MCP tools/call by buffering the Plexus stream
    pub async fn handle_tools_call(&self, params: ToolsCallParams) -> Result<ToolsCallResult> {
        self.require_state(McpState::Ready)?;

        // Parse tool name
        let (namespace, method) = params.name.split_once('.')
            .ok_or_else(|| JsonRpcError::invalid_params("Invalid tool name"))?;

        // Get stream from Plexus (same as native clients would)
        let stream = self.plexus.call(
            &format!("{}.{}", namespace, method),
            params.arguments
        ).await?;

        // Buffer and return
        Ok(Self::buffer_stream(stream).await)
    }
}
```

### Comparison: MCP vs Native Call

```rust
// === MCP Client (Claude Code) ===
// Request:
{ "method": "tools/call", "params": { "name": "bash.execute", "arguments": { "command": "ls" }}}
// Response (after buffering):
{ "result": { "content": [{ "type": "text", "text": "file1.txt\nfile2.txt" }], "isError": false }}

// === Native Client (Symbols) ===
// Request:
{ "method": "bash.execute", "params": ["ls"] }
// Response (subscription ID):
{ "result": 12345 }
// Stream events:
{ "method": "bash.execute", "params": { "subscription": 12345, "result": { "type": "start" }}}
{ "method": "bash.execute", "params": { "subscription": 12345, "result": { "type": "stdout", "data": "file1.txt\n" }}}
{ "method": "bash.execute", "params": { "subscription": 12345, "result": { "type": "stdout", "data": "file2.txt\n" }}}
{ "method": "bash.execute", "params": { "subscription": 12345, "result": { "type": "complete", "exit_code": 0 }}}
```

### Trade-offs

| Aspect | MCP (Buffered) | Plexus (Streaming) |
|--------|----------------|-------------------|
| Latency to first byte | High (waits for completion) | Low (immediate) |
| Memory usage | Buffers entire response | Constant (event-by-event) |
| Cancellation | Limited (must complete) | Full support |
| Protocol compliance | MCP spec compliant | Native Plexus |
| Client complexity | Simple (single response) | Complex (subscription mgmt) |

### Future: Optional Progress Notifications

MCP supports optional progress notifications for long-running operations:

```json
{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "abc123",
    "progress": 50,
    "total": 100
  }
}
```

This could bridge the gap - stream events could trigger progress notifications while still buffering the final response:

```rust
async fn buffer_stream_with_progress(
    stream: impl Stream<Item = ActivationEvent>,
    progress_token: Option<String>,
    notify: impl Fn(Notification),
) -> ToolsCallResult {
    let mut event_count = 0;

    pin_mut!(stream);
    while let Some(event) = stream.next().await {
        event_count += 1;

        // Send progress notification if token provided
        if let Some(ref token) = progress_token {
            notify(Notification::Progress {
                progress_token: token.clone(),
                progress: event_count,
                total: None, // Unknown total
            });
        }

        // ... buffer as before
    }

    // Return buffered result
    ToolsCallResult { ... }
}
```

## Unbuffered Streaming Implementation (Preferred)

When using Streamable HTTP transport, we can stream Plexus events directly as MCP progress notifications without buffering.

### Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                  Unbuffered MCP Streaming                        │
│                                                                  │
│  tools/call request                                              │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  1. Parse tool name, create progress token               │    │
│  └─────────────────────────────────────────────────────────┘    │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  2. Call Plexus activation (returns Stream<Event>)       │    │
│  └─────────────────────────────────────────────────────────┘    │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  3. Stream: Forward events as SSE                        │    │
│  │                                                          │    │
│  │     Plexus Event              SSE Output                 │    │
│  │     ────────────              ──────────                 │    │
│  │     Start { ... }     →       (skip or progress msg)     │    │
│  │     Content { "Hi" }  →       notifications/progress     │    │
│  │                               { message: "Hi" }          │    │
│  │     ToolUse { ... }   →       notifications/progress     │    │
│  │                               { message: "Using X..." }  │    │
│  │     Complete { ... }  →       tools/call result          │    │
│  │                               { content: [...] }         │    │
│  │                                                          │    │
│  └─────────────────────────────────────────────────────────┘    │
│       │                                                          │
│       ▼                                                          │
│  SSE stream (no buffering, immediate delivery)                   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

### Implementation

```rust
/// Unbuffered MCP streaming - preferred mode
impl McpInterface {
    /// Stream Plexus events directly as MCP SSE events
    fn stream_tools_call(
        &self,
        request_id: Value,
        stream: impl Stream<Item = ActivationEvent>,
    ) -> impl Stream<Item = SseEvent> {
        let progress_token = Uuid::new_v4().to_string();
        let mut event_id = 0;
        let mut final_content = Vec::new();

        async_stream::stream! {
            pin_mut!(stream);

            while let Some(event) = stream.next().await {
                event_id += 1;

                match event {
                    // Stream content as progress notifications
                    ActivationEvent::Content { text } => {
                        final_content.push(McpContent::Text { text: text.clone() });

                        yield SseEvent {
                            id: format!("evt-{}", event_id),
                            data: json!({
                                "jsonrpc": "2.0",
                                "method": "notifications/progress",
                                "params": {
                                    "progressToken": progress_token,
                                    "progress": event_id,
                                    "message": text  // 2025-03-26: new message field
                                }
                            })
                        };
                    }

                    // Stream tool usage as progress
                    ActivationEvent::ToolUse { tool_name, .. } => {
                        yield SseEvent {
                            id: format!("evt-{}", event_id),
                            data: json!({
                                "jsonrpc": "2.0",
                                "method": "notifications/progress",
                                "params": {
                                    "progressToken": progress_token,
                                    "progress": event_id,
                                    "message": format!("Using tool: {}", tool_name)
                                }
                            })
                        };
                    }

                    // Final result - aggregate and send
                    ActivationEvent::Complete { .. } => {
                        yield SseEvent {
                            id: format!("evt-{}", event_id),
                            data: json!({
                                "jsonrpc": "2.0",
                                "id": request_id,
                                "result": {
                                    "content": final_content,
                                    "isError": false
                                }
                            })
                        };
                        break;
                    }

                    // Errors
                    ActivationEvent::Error { message } => {
                        yield SseEvent {
                            id: format!("evt-{}", event_id),
                            data: json!({
                                "jsonrpc": "2.0",
                                "id": request_id,
                                "result": {
                                    "content": [{ "type": "text", "text": message }],
                                    "isError": true
                                }
                            })
                        };
                        break;
                    }

                    _ => {}
                }
            }
        }
    }
}
```

### HTTP Response

```http
POST /mcp HTTP/1.1
Content-Type: application/json
Accept: application/json, text/event-stream

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"claudecode.chat","arguments":{...}}}
```

```http
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Mcp-Session-Id: abc123

event: message
id: evt-1
data: {"jsonrpc":"2.0","method":"notifications/progress","params":{"progressToken":"xyz","progress":1,"message":"Hello"}}

event: message
id: evt-2
data: {"jsonrpc":"2.0","method":"notifications/progress","params":{"progressToken":"xyz","progress":2,"message":" world"}}

event: message
id: evt-3
data: {"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"Hello world"}],"isError":false}}
```

### Comparison: Unbuffered vs Buffered

| Aspect | Unbuffered (Streamable HTTP) | Buffered (stdio/simple HTTP) |
|--------|------------------------------|------------------------------|
| Latency to first byte | **Immediate** | Waits for completion |
| Memory usage | **Constant** | Grows with response |
| Real-time feedback | **Yes** (progress notifications) | No |
| Client complexity | Must handle SSE | Simple JSON response |
| Protocol version | 2025-03-26 | 2024-11-05 compatible |
| Transport | HTTP with SSE | stdio or simple HTTP |

### When to Use Each Mode

```rust
fn handle_tools_call(&self, req: HttpRequest, params: ToolsCallParams) -> Response {
    // Check Accept header for SSE support
    let accepts_sse = req.headers()
        .get("Accept")
        .map(|v| v.contains("text/event-stream"))
        .unwrap_or(false);

    if accepts_sse {
        // Unbuffered streaming mode
        let stream = self.plexus.call(&params.name, params.arguments).await;
        Response::sse(self.stream_tools_call(req.id, stream))
    } else {
        // Buffered mode (fallback)
        let stream = self.plexus.call(&params.name, params.arguments).await;
        let result = Self::buffer_stream(stream).await;
        Response::json(result)
    }
}
```

---

## Capability Mapping

| Plexus Activation | MCP Capability | Details |
|-------------------|----------------|---------|
| Any activation | `tools` | All methods exposed as tools |
| Arbor | `resources` | Trees/nodes as resources |
| Cone (optional) | `prompts` | Sessions as prompt templates |
| Tracing | `logging` | Log messages via notifications |

## Implementation Checklist

### Phase 1: Core Protocol
- [ ] Add `McpState` enum and state machine
- [ ] Implement `initialize` handler with capability building
- [ ] Implement `notifications/initialized` handler
- [ ] Implement `ping` handler
- [ ] Guard all other methods with state check

### Phase 2: Tools
- [ ] Implement `tools/list` with schema transformation
- [ ] Implement `tools/call` with stream collection
- [ ] Handle tool errors (isError: true)
- [ ] Add pagination support

### Phase 3: Resources (Optional)
- [ ] Design Arbor → Resource URI scheme
- [ ] Implement `resources/list`
- [ ] Implement `resources/read`
- [ ] Implement `resources/subscribe` (if needed)

### Phase 4: Utilities
- [ ] Implement `notifications/cancelled`
- [ ] Add request ID tracking for cancellation
- [ ] Implement `logging/setLevel` (optional)

### Phase 5: Validation
- [ ] Run mcp-validator test suite
- [ ] Test with Claude Code as client
- [ ] Document any spec deviations

## Validation Command

```bash
# Run MCP compliance tests
cd /tmp/mcp-validator
source .venv/bin/activate
python -m mcp_testing.scripts.compliance_report \
  --server-command "/path/to/substrate --stdio --mcp" \
  --protocol-version 2024-11-05 \
  --test-timeout 30
```

## CLI Flag

```bash
# Current behavior (Plexus JSON-RPC, no MCP handshake)
substrate --stdio

# MCP-compatible mode (adds initialize requirement)
substrate --stdio --mcp
```

## References

### MCP 2025-03-26 (Current)
- [MCP Specification 2025-03-26](https://modelcontextprotocol.io/specification/2025-03-26)
- [Streamable HTTP Transport](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports#streamable-http)
- [Key Changes from 2024-11-05](https://modelcontextprotocol.io/specification/2025-03-26/changelog)
- [Tools](https://modelcontextprotocol.io/specification/2025-03-26/server/tools)

### MCP 2024-11-05 (Legacy)
- [MCP Specification 2024-11-05](https://modelcontextprotocol.io/specification/2024-11-05)
- [Lifecycle](https://modelcontextprotocol.io/specification/2024-11-05/basic/lifecycle)
- [HTTP+SSE Transport (deprecated)](https://modelcontextprotocol.io/specification/2024-11-05/basic/transports)

### Tools
- [mcp-validator](https://github.com/Janix-ai/mcp-validator)
- [MCP Inspector](https://github.com/modelcontextprotocol/inspector)
