# Compiler Architecture: From IR to Typed Clients

**Status**: Design
**Scope**: Full-stack client generation pipeline
**Updated**: 2025-12-30

## Executive Summary

This document outlines the architecture for generating fully-typed clients from the Plexus IR. The goal is a **fully automatic** pipeline where changes to Rust backend types propagate to generated clients without manual intervention.

```
Rust types → hub-macro → Schema → IR → Transpiler → TypeScript/Python/Swift
```

The generated client should feel **native** - no string method names, full autocomplete, namespaces that mirror the hub structure.

## Design Principles

### 1. Everything is Streaming

All methods return streams. The protocol is uniform:
- "Non-streaming" methods emit one event then complete
- "Streaming" methods emit multiple events then complete
- Errors are events in the stream (`{ type: "error", message: "..." }`)

No distinction at the protocol level. The `streaming` flag tells **consumers** whether to expect one event or many.

**Call flow (identical for both):**

```
Client                              Server
  │                                   │
  ├─ JSON-RPC ───────────────────────►│
  │  { method: "plexus_call",         │
  │    params: { method: "cone.create", params: {...} } }
  │                                   │
  │                         plexus.route("cone.create", params)
  │                                   │
  │                         // Generated by hub-macro:
  │                         let stream = self.create(...).await;
  │                         Ok(wrap_stream(stream, "cone.create", [...]))
  │                                   │
  │◄── StreamItem::Data ──────────────┤  { type: "cone_created", ... }
  │◄── StreamItem::Done ──────────────┤
```

Both "sync" and "streaming" methods use the same flow. The Rust method always returns `impl Stream<Item = T>`. For "sync" methods, the stream yields exactly one item:

```rust
async fn create(...) -> impl Stream<Item = CreateResult> {
    stream! {
        yield CreateResult::ConeCreated { cone_id, head };  // ONE item
    }
}

async fn chat(...) -> impl Stream<Item = ChatEvent> {
    stream! {
        yield ChatEvent::ChatStart { ... };      // MULTIPLE items
        yield ChatEvent::ChatContent { ... };
        yield ChatEvent::ChatComplete { ... };
    }
}
```

### 2. Streaming Declared in Schema (Not Inferred)

No explicit `#[streaming]` attribute needed in user code. The **hub-macro already knows** from the Rust return type:

```rust
// Macro sees: impl Stream<Item = ChatEvent> → streaming: true
async fn chat(...) -> impl Stream<Item = ChatEvent>

// Macro sees: impl Stream<Item = CreateResult> → streaming: false
async fn create(...) -> impl Stream<Item = CreateResult>
```

The macro's `extract_stream_item_type()` already detects this. What's missing is **emitting it in the schema**. The `MethodSchema` struct needs a `streaming: bool` field that the macro populates.

**Current gap**: The IR builder infers streaming from return type variant count (heuristic). This breaks for shared mega-unions like `ArborEvent`. The fix is explicit declaration, not better inference.

### 3. WebSocket + JSON-RPC Only

Single transport, single protocol. No negotiation complexity:
- Transport: WebSocket
- Protocol: JSON-RPC 2.0 with subscriptions
- All calls go through `plexus_call` method

### 4. Native Feel

Generated clients should mirror the hub structure exactly:
```typescript
// Not this (stringly typed)
client.call("cone.chat", { ... })

// This (fully typed, native feel)
hub.cone.chat({ identifier: { by_name: "x" }, prompt: "hi" })
```

## Current State

### What We Have

```
┌─────────────────────────────────────────────────────────────┐
│ Plexus (Rust)                                               │
│ ┌─────────────┐    ┌─────────────┐    ┌─────────────┐      │
│ │ Rust Types  │───▶│ hub-macro   │───▶│ JSON Schema │      │
│ │ + schemars  │    │             │    │ per method  │      │
│ └─────────────┘    └─────────────┘    └─────────────┘      │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Synapse (Haskell)                                           │
│ ┌─────────────┐    ┌─────────────┐    ┌─────────────┐      │
│ │ Schema Walk │───▶│ IR Builder  │───▶│ JSON IR     │      │
│ │ (hyloM)     │    │             │    │ (emitted)   │      │
│ └─────────────┘    └─────────────┘    └─────────────┘      │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Transpiler (TO BE BUILT)                                    │
│ ┌─────────────┐    ┌─────────────┐    ┌─────────────┐      │
│ │ IR Parser   │───▶│ Code Gen    │───▶│ TS/Py/Swift │      │
│ │             │    │             │    │ Client      │      │
│ └─────────────┘    └─────────────┘    └─────────────┘      │
└─────────────────────────────────────────────────────────────┘
```

## Required Protocol Changes

Before building the transpiler, the schema must explicitly declare streaming:

### 1. Add `streaming` to MethodSchema

```rust
// substrate/src/plexus/schema.rs
pub struct MethodSchema {
    pub name: String,
    pub description: String,
    pub hash: String,
    pub params: Option<schemars::Schema>,
    pub returns: Option<schemars::Schema>,
    pub streaming: bool,  // ← ADD THIS
}

impl MethodSchema {
    pub fn with_streaming(mut self, streaming: bool) -> Self {
        self.streaming = streaming;
        self
    }
}
```

### 2. Emit streaming flag from hub-macro

```rust
// hub-macro/src/codegen/method_enum.rs
// In method_schemas() generation:

let is_streaming = m.stream_item_type.is_some();

schema = schema
    .with_params(p)
    .with_returns(r)
    .with_streaming(is_streaming);  // ← ADD THIS
```

The macro already knows via `stream_item_type: Option<Type>`. This change is purely additive.

### 3. Update Synapse to read the flag

```haskell
-- synapse/src/Synapse/Schema/Types.hs
data MethodSchema = MethodSchema
  { methodName        :: Text
  , methodDescription :: Text
  , methodHash        :: Text
  , methodParams      :: Maybe Value
  , methodReturns     :: Maybe Value
  , methodStreaming   :: Bool  -- ← ADD THIS
  }

-- synapse/src/Synapse/IR/Builder.hs
-- Replace inference with explicit read:
let streaming = methodStreaming method  -- not inferred
```

### 4. Fix arbor's shared mega-union (design debt)

Current arbor uses one `ArborEvent` for all 19 methods:
```rust
fn tree_create(...) -> ArborEvent  // 30 variants
fn tree_get(...) -> ArborEvent     // same 30 variants
```

Should be per-method result types:
```rust
fn tree_create(...) -> TreeCreateResult  // tree_created | error
fn tree_get(...) -> TreeGetResult        // tree_data | error
```

This is independent of the streaming flag but improves schema clarity.

---

### What the IR Provides

The current IR already has everything needed for type generation:

```json
{
  "irVersion": "1.0",
  "irPlugins": {
    "cone": ["create", "get", "list", "delete", "chat", "set_head", "registry"]
  },
  "irTypes": {
    "Position": { "tdKind": { "tag": "KindStruct", "ksFields": [...] } },
    "ChatEvent": { "tdKind": { "tag": "KindEnum", "keVariants": [...] } }
  },
  "irMethods": {
    "cone.chat": {
      "mdStreaming": true,
      "mdParams": [...],
      "mdReturns": { "tag": "RefNamed", "contents": "ChatEvent" }
    }
  }
}
```

## Two-Layer Client Architecture

### Layer 1: Transport (Internal)

Low-level, handles WebSocket + JSON-RPC:

```typescript
// Internal - not exposed to users
class HubTransport {
  private ws: WebSocket;
  private nextId = 1;
  private subscriptions = new Map<number, StreamController>();

  async *stream<T>(method: string, params: unknown): AsyncGenerator<T> {
    const id = this.nextId++;
    const controller = new StreamController<T>();
    this.subscriptions.set(id, controller);

    this.ws.send(JSON.stringify({
      jsonrpc: "2.0",
      id,
      method: "plexus_call",
      params: { method, params }
    }));

    try {
      yield* controller;
    } finally {
      this.subscriptions.delete(id);
    }
  }
}
```

### Layer 2: Typed Namespaces (Public API)

High-level, mirrors hub structure:

```typescript
// Public API - what users interact with
export interface Hub {
  readonly cone: ConeNamespace;
  readonly arbor: ArborNamespace;
  readonly health: HealthNamespace;
}

export interface ConeNamespace {
  create(params: CreateParams): Promise<CreateResult>;
  get(params: GetParams): Promise<GetResult>;
  list(): Promise<ListResult>;
  delete(params: DeleteParams): Promise<DeleteResult>;
  chat(params: ChatParams): AsyncGenerator<ChatEvent>;
  set_head(params: SetHeadParams): Promise<SetHeadResult>;
  registry(): Promise<RegistryResult>;
}
```

### Return Type Mapping

The `mdStreaming` flag determines the return type:

| `mdStreaming` | Return Type | Behavior |
|---------------|-------------|----------|
| `false` | `Promise<T>` | Collects single event from stream |
| `true` | `AsyncGenerator<T>` | Yields all events from stream |

```typescript
// Generated implementation
function createConeNamespace(transport: HubTransport): ConeNamespace {
  return {
    // mdStreaming: false → Promise (collect first event)
    create: (params) => collect(transport.stream("cone.create", params)),
    get: (params) => collect(transport.stream("cone.get", params)),
    list: () => collect(transport.stream("cone.list", {})),

    // mdStreaming: true → AsyncGenerator (pass through)
    chat: (params) => transport.stream("cone.chat", params),
  };
}

// Helper to collect single-event streams
async function collect<T>(stream: AsyncGenerator<T>): Promise<T> {
  for await (const event of stream) {
    return event;
  }
  throw new Error("Stream completed without events");
}
```

## Generated Output Structure

```
generated/
├── types.ts           # All type definitions
├── transport.ts       # HubTransport class (internal)
├── namespaces/
│   ├── cone.ts        # ConeNamespace interface + implementation
│   ├── arbor.ts       # ArborNamespace interface + implementation
│   └── index.ts       # Re-exports
├── hub.ts             # Hub interface + createHub factory
└── index.ts           # Public exports
```

## Type Generation

### Structs

**Input (IR):**
```json
{
  "Position": {
    "tdDescription": "A position in the context tree",
    "tdKind": {
      "tag": "KindStruct",
      "ksFields": [
        { "fdName": "tree_id", "fdType": { "tag": "RefNamed", "contents": "UUID" }, "fdDescription": "The tree containing this position" },
        { "fdName": "node_id", "fdType": { "tag": "RefNamed", "contents": "UUID" }, "fdDescription": "The specific node within the tree" }
      ]
    }
  }
}
```

**Output (TypeScript):**
```typescript
/** A position in the context tree */
export interface Position {
  /** The tree containing this position */
  tree_id: string;
  /** The specific node within the tree */
  node_id: string;
}
```

### Discriminated Unions

**Input (IR):**
```json
{
  "ChatEvent": {
    "tdDescription": "Events emitted during cone.chat (streaming)",
    "tdKind": {
      "tag": "KindEnum",
      "keDiscriminator": "type",
      "keVariants": [
        { "vdName": "chat_start", "vdFields": [...] },
        { "vdName": "chat_content", "vdFields": [...] },
        { "vdName": "chat_complete", "vdFields": [...] },
        { "vdName": "error", "vdFields": [...] }
      ]
    }
  }
}
```

**Output (TypeScript):**
```typescript
/** Events emitted during cone.chat (streaming) */
export type ChatEvent =
  | ChatEvent_ChatStart
  | ChatEvent_ChatContent
  | ChatEvent_ChatComplete
  | ChatEvent_Error;

export interface ChatEvent_ChatStart {
  readonly type: "chat_start";
  readonly cone_id: string;
  readonly user_position: Position;
}

export interface ChatEvent_ChatContent {
  readonly type: "chat_content";
  readonly cone_id: string;
  readonly content: string;
}

export interface ChatEvent_ChatComplete {
  readonly type: "chat_complete";
  readonly cone_id: string;
  readonly new_head: Position;
  readonly usage?: ChatUsage | null;
}

export interface ChatEvent_Error {
  readonly type: "error";
  readonly message: string;
}

// Type guards for narrowing
export function isChatStart(e: ChatEvent): e is ChatEvent_ChatStart {
  return e.type === "chat_start";
}

export function isChatContent(e: ChatEvent): e is ChatEvent_ChatContent {
  return e.type === "chat_content";
}

export function isChatComplete(e: ChatEvent): e is ChatEvent_ChatComplete {
  return e.type === "chat_complete";
}

export function isChatError(e: ChatEvent): e is ChatEvent_Error {
  return e.type === "error";
}
```

## Namespace Generation

For each plugin in `irPlugins`, generate:

```typescript
// generated/namespaces/cone.ts

import { HubTransport, collect } from "../transport";
import {
  CreateParams, CreateResult,
  GetParams, GetResult,
  ChatParams, ChatEvent,
  // ...
} from "../types";

export interface ConeNamespace {
  /** Create a new cone (LLM agent with persistent conversation context) */
  create(params: CreateParams): Promise<CreateResult>;

  /** Get cone configuration by name or ID */
  get(params: GetParams): Promise<GetResult>;

  /** List all cones */
  list(): Promise<ListResult>;

  /** Delete a cone (associated tree is preserved) */
  delete(params: DeleteParams): Promise<DeleteResult>;

  /** Chat with a cone - appends prompt to context, calls LLM, advances head */
  chat(params: ChatParams): AsyncGenerator<ChatEvent>;

  /** Move cone's canonical head to a different node in the tree */
  set_head(params: SetHeadParams): Promise<SetHeadResult>;

  /** Get available LLM services and models */
  registry(): Promise<RegistryResult>;
}

export function createConeNamespace(transport: HubTransport): ConeNamespace {
  return {
    create: (params) => collect(transport.stream<CreateResult>("cone.create", params)),
    get: (params) => collect(transport.stream<GetResult>("cone.get", params)),
    list: () => collect(transport.stream<ListResult>("cone.list", {})),
    delete: (params) => collect(transport.stream<DeleteResult>("cone.delete", params)),
    chat: (params) => transport.stream<ChatEvent>("cone.chat", params),
    set_head: (params) => collect(transport.stream<SetHeadResult>("cone.set_head", params)),
    registry: () => collect(transport.stream<RegistryResult>("cone.registry", {})),
  };
}
```

## Hub Factory

```typescript
// generated/hub.ts

import { HubTransport } from "./transport";
import { ConeNamespace, createConeNamespace } from "./namespaces/cone";
import { ArborNamespace, createArborNamespace } from "./namespaces/arbor";
import { HealthNamespace, createHealthNamespace } from "./namespaces/health";

export interface Hub {
  readonly cone: ConeNamespace;
  readonly arbor: ArborNamespace;
  readonly health: HealthNamespace;
}

export interface HubOptions {
  url: string;
  onError?: (error: Error) => void;
  onDisconnect?: () => void;
}

export async function createHub(options: HubOptions): Promise<Hub> {
  const transport = new HubTransport(options);
  await transport.connect();

  return {
    cone: createConeNamespace(transport),
    arbor: createArborNamespace(transport),
    health: createHealthNamespace(transport),
  };
}

// Metadata for version checking
export const HUB_SCHEMA_HASH = "449f272b6a97dd94";
```

## Usage Examples

### Basic Usage

```typescript
import { createHub, isChatContent, isChatComplete } from "./generated";

const hub = await createHub({ url: "ws://localhost:4444" });

// Create a cone - returns Promise (single event)
const created = await hub.cone.create({
  name: "my-agent",
  model_id: "claude-3-haiku"
});

if (created.type === "cone_created") {
  console.log("Created:", created.cone_id);
} else {
  console.error("Error:", created.message);
}

// Chat - returns AsyncGenerator (multiple events)
for await (const event of hub.cone.chat({
  identifier: { by_name: "my-agent" },
  prompt: "Hello!"
})) {
  if (isChatContent(event)) {
    process.stdout.write(event.content);
  } else if (isChatComplete(event)) {
    console.log("\nTokens:", event.usage?.total_tokens);
  } else if (event.type === "error") {
    console.error("Error:", event.message);
  }
}
```

### Error Handling

Errors are events in the stream, handled by pattern matching:

```typescript
const result = await hub.cone.get({ identifier: { by_name: "nonexistent" } });

switch (result.type) {
  case "cone_data":
    console.log("Found:", result.cone.name);
    break;
  case "error":
    console.error("Not found:", result.message);
    break;
}
```

### Nested Plugins

For hub plugins that contain other plugins:

```typescript
// If solar is a hub containing mercury, venus, etc.
const mercuryInfo = await hub.solar.mercury.info();
```

## Transport Implementation

```typescript
// generated/transport.ts

type StreamController<T> = {
  push(value: T): void;
  complete(): void;
  error(err: Error): void;
  [Symbol.asyncIterator](): AsyncIterator<T>;
};

export class HubTransport {
  private ws!: WebSocket;
  private nextId = 1;
  private subscriptions = new Map<number, StreamController<unknown>>();
  private connected = false;

  constructor(private options: { url: string; onError?: (e: Error) => void }) {}

  async connect(): Promise<void> {
    return new Promise((resolve, reject) => {
      this.ws = new WebSocket(this.options.url);
      this.ws.onopen = () => {
        this.connected = true;
        resolve();
      };
      this.ws.onerror = (e) => reject(new Error("Connection failed"));
      this.ws.onmessage = this.handleMessage.bind(this);
    });
  }

  private handleMessage(event: MessageEvent) {
    const msg = JSON.parse(event.data);

    // JSON-RPC subscription notification
    if (msg.method === "subscription" && msg.params) {
      const { subscription: id, result } = msg.params;
      const controller = this.subscriptions.get(id);
      if (!controller) return;

      const data = result.content;  // PlexusStreamItem.content

      if (result.type === "done") {
        controller.complete();
      } else if (result.type === "error") {
        controller.push(data);  // Error is an event, not an exception
      } else if (result.type === "data") {
        controller.push(data);
      }
    }
  }

  async *stream<T>(method: string, params: unknown): AsyncGenerator<T> {
    const id = this.nextId++;
    const controller = createStreamController<T>();
    this.subscriptions.set(id, controller as StreamController<unknown>);

    this.ws.send(JSON.stringify({
      jsonrpc: "2.0",
      id,
      method: "plexus_call",
      params: { method, params: params ?? {} }
    }));

    try {
      yield* controller;
    } finally {
      this.subscriptions.delete(id);
    }
  }
}

function createStreamController<T>(): StreamController<T> {
  const queue: T[] = [];
  let resolve: (() => void) | null = null;
  let done = false;

  return {
    push(value: T) {
      queue.push(value);
      resolve?.();
    },
    complete() {
      done = true;
      resolve?.();
    },
    error(err: Error) {
      // For now, errors come through as events
      done = true;
      resolve?.();
    },
    async *[Symbol.asyncIterator]() {
      while (!done || queue.length > 0) {
        if (queue.length > 0) {
          yield queue.shift()!;
        } else if (!done) {
          await new Promise<void>(r => resolve = r);
        }
      }
    }
  };
}

export async function collect<T>(stream: AsyncGenerator<T>): Promise<T> {
  for await (const event of stream) {
    return event;
  }
  throw new Error("Stream completed without events");
}
```

## Transpiler CLI

```typescript
// hub-codegen/src/cli.ts
import { program } from "commander";
import { generate } from "./generator";

program
  .name("hub-codegen")
  .description("Generate typed clients from Hub IR")
  .argument("<ir-file>", "Path to IR JSON file (or - for stdin)")
  .option("-o, --output <dir>", "Output directory", "./generated")
  .option("-l, --language <lang>", "Target language", "typescript")
  .action(async (irFile, options) => {
    const irJson = irFile === "-"
      ? await readStdin()
      : await fs.readFile(irFile, "utf-8");

    const ir = JSON.parse(irJson);
    await generate(ir, options.output, options.language);

    console.log(`Generated ${options.language} client in ${options.output}`);
  });

program.parse();
```

## Full Pipeline

```bash
# 1. Start hub
cargo run --bin substrate

# 2. Generate IR from live hub
synapse --emit-ir > ir.json

# 3. Generate TypeScript client
hub-codegen ir.json -o ./src/generated

# 4. Use in your app
import { createHub } from "./generated";

const hub = await createHub({ url: "ws://localhost:4444" });
const result = await hub.cone.create({ name: "test", model_id: "gpt-4" });
```

Or as a one-liner during development:

```bash
synapse --emit-ir | hub-codegen - -o ./src/generated
```

## Reliability

### Schema Hash Verification

```typescript
import { createHub, HUB_SCHEMA_HASH } from "./generated";

const hub = await createHub({ url });

// Verify schema hasn't changed
const serverHash = await hub.plexus.hash();
if (serverHash !== HUB_SCHEMA_HASH) {
  console.warn("Schema mismatch - regenerate client");
}
```

### Type Guards

Generated type guards enable safe narrowing:

```typescript
for await (const event of hub.cone.chat(params)) {
  if (isChatContent(event)) {
    // event is narrowed to ChatEvent_ChatContent
    console.log(event.content);  // ✓ TypeScript knows this exists
  }
}
```

## Future Extensions

### Optional Enhancements

1. **Runtime validation** - Generate zod schemas for runtime type checking
2. **Branded types** - `UUID` as `string & { __brand: "UUID" }` for extra safety
3. **Reconnection** - Auto-reconnect with subscription replay
4. **Request queuing** - Queue requests while disconnected

### Additional Languages

The same IR can generate clients for:
- **Python** - asyncio + websockets, dataclasses or Pydantic
- **Swift** - async/await, Codable structs
- **Go** - goroutines, struct types

## Success Criteria

### Protocol (Prerequisites)
- [x] IR captures all necessary type information
- [ ] `streaming: bool` added to `MethodSchema` in substrate
- [ ] hub-macro emits streaming flag (it already knows)
- [ ] Synapse reads streaming flag (removes inference heuristic)
- [ ] Arbor uses per-method return types (design debt, can parallelize)

### Transpiler
- [ ] Generated TypeScript compiles without errors
- [ ] Namespace structure mirrors hub structure
- [ ] `Promise<T>` for single-event methods (`mdStreaming: false`)
- [ ] `AsyncGenerator<T>` for streaming methods (`mdStreaming: true`)
- [ ] Type guards narrow discriminated unions
- [ ] Schema hash enables version checking
- [ ] Zero manual edits needed in generated code
