# IR Codegen Chain: Full Pipeline Architecture

**Status**: Active
**Scope**: End-to-end typed client generation from Rust to TypeScript
**Updated**: 2025-01-03
**Linked**: substrate, synapse, hub-codegen

## Overview

The IR codegen chain transforms Rust type definitions into fully-typed TypeScript clients through a multi-stage pipeline:

```
                    RUST                        SYNAPSE                    HUB-CODEGEN
              ┌─────────────────┐         ┌─────────────────┐         ┌─────────────────┐
              │                 │         │                 │         │                 │
Rust Types ──>│   hub-macro     │──JSON──>│   IR Builder    │──JSON──>│   Generators    │──> TypeScript
              │   (schemars)    │ Schema  │   (hyloM walk)  │   IR    │   (per-file)    │    Client
              │                 │         │                 │         │                 │
              └─────────────────┘         └─────────────────┘         └─────────────────┘
                     │                           │                           │
                     ▼                           ▼                           ▼
              substrate serves           synapse substrate -i           types.ts, transport.ts
              plexus.schema              emits IR JSON               rpc.ts, namespaces/*
```

## Pipeline Stages

### Stage 1: Rust Types -> JSON Schema (substrate)

**Location**: `hub-macro/src/codegen/`

The `#[hub_method]` macro generates JSON Schema from Rust types using schemars:

```rust
// Input: Rust method definition
#[hub_method]
async fn chat(&self, params: ChatParams) -> impl Stream<Item = ChatEvent> { ... }

// Output: JSON Schema (served via plexus.schema RPC)
{
  "method": "chat",
  "params": { "$defs": {...}, "properties": {...}, "required": [...] },
  "returns": { "title": "ChatEvent", "oneOf": [...] }
}
```

**Key files:**
- `hub-macro/src/codegen/method_enum.rs` - Schema generation
- `hub-macro/src/codegen/plugin.rs` - Plugin trait implementation
- `substrate/src/plexus/plexus.rs` - Schema serving via RPC

### Stage 2: JSON Schema -> IR (synapse)

**Location**: `synapse/src/Synapse/IR/`

Synapse walks the schema tree via `plexus.schema` RPC, building a deduplicated IR:

```bash
# Run introspection mode
synapse substrate -i

# Outputs JSON IR to stdout
{
  "irVersion": "1.0",
  "irHash": "449f272b6a97dd94",
  "irTypes": { ... },
  "irMethods": { ... },
  "irPlugins": { ... }
}
```

**Key files:**
- `synapse/src/Synapse/IR/Types.hs` - IR data types
- `synapse/src/Synapse/IR/Builder.hs` - Schema -> IR transformation
- `synapse/src/Synapse/Algebra/Walk.hs` - hyloM-based tree walking

**IR Structure:**

```haskell
data IR = IR
  { irVersion :: Text                    -- "1.0"
  , irHash    :: Maybe Text              -- Plexus hash for cache invalidation
  , irTypes   :: Map Text TypeDef        -- Deduplicated type definitions
  , irMethods :: Map Text MethodDef      -- Methods keyed by "plugin.method"
  , irPlugins :: Map Text [Text]         -- Plugin -> method name mapping
  }

data TypeDef = TypeDef
  { tdName        :: Text
  , tdDescription :: Maybe Text
  , tdKind        :: TypeKind  -- KindStruct | KindEnum | KindAlias | KindPrimitive
  }

data MethodDef = MethodDef
  { mdName        :: Text       -- "chat"
  , mdFullPath    :: Text       -- "cone.chat"
  , mdNamespace   :: Text       -- "cone"
  , mdStreaming   :: Bool       -- Inferred from return type
  , mdParams      :: [ParamDef]
  , mdReturns     :: TypeRef
  }
```

### Stage 3: IR -> TypeScript (hub-codegen)

**Location**: `hub-codegen/src/`

Rust-based code generator consumes IR JSON and emits TypeScript:

```bash
# Generate TypeScript client
synapse substrate -i | cargo run --manifest-path hub-codegen/Cargo.toml -- -o /tmp/client

# Output structure:
/tmp/client/
  types.ts        # All type definitions
  transport.ts    # WebSocket + JSON-RPC layer
  rpc.ts          # Method wrapper functions
  namespaces/     # Per-plugin namespace exports
    cone.ts
    arbor.ts
    ...
  index.ts        # Public API
  package.json    # npm package metadata
```

**Key files:**
- `hub-codegen/src/ir.rs` - IR deserialization (mirrors Synapse types)
- `hub-codegen/src/generator/types.ts` - Type generation
- `hub-codegen/src/generator/transport.ts` - Transport layer
- `hub-codegen/src/generator/rpc.ts` - RPC method wrappers
- `hub-codegen/src/generator/namespaces.rs` - Namespace organization
- `hub-codegen/src/generator/package.rs` - package.json generation

## Running the Full Pipeline

```bash
# Prerequisites: substrate running
cargo run --bin substrate &

# One-liner for development
synapse substrate -i | cargo run --manifest-path hub-codegen/Cargo.toml -- -o /tmp/client

# Or step by step:
synapse substrate -i > ir.json
cargo run --manifest-path hub-codegen/Cargo.toml -- ir.json -o /tmp/client
```

## IR Type System

### TypeRef (type references)

```haskell
data TypeRef
  = RefNamed Text                   -- Reference to named type: "Position"
  | RefPrimitive Text (Maybe Text)  -- Primitive with format: ("string", Just "uuid")
  | RefArray TypeRef                -- Array: string[]
  | RefOptional TypeRef             -- Nullable: T | null
  | RefAny                          -- Intentionally dynamic (serde_json::Value)
  | RefUnknown                      -- Schema gap (should warn)
```

### TypeKind (type shapes)

```haskell
data TypeKind
  = KindStruct [FieldDef]              -- { field: Type, ... }
  | KindEnum Text [VariantDef]         -- Discriminated union (discriminator, variants)
  | KindAlias TypeRef                  -- type Foo = Bar
  | KindPrimitive Text (Maybe Text)    -- string, number, boolean with format
```

### TypeScript Mapping

| IR TypeRef | TypeScript |
|------------|------------|
| `RefNamed "Position"` | `Position` |
| `RefPrimitive "string" (Just "uuid")` | `string` |
| `RefPrimitive "integer" _` | `number` |
| `RefArray inner` | `inner[]` |
| `RefOptional inner` | `inner \| null` |
| `RefAny` | `unknown` |
| `RefUnknown` | `unknown` (with warning) |

## Known Limitations

### 1. Streaming Inference Heuristic

Currently, `mdStreaming` is inferred from return type structure:

```haskell
-- Builder.hs
inferStreaming :: TypeDef -> Bool
inferStreaming td = case tdKind td of
  KindEnum _ variants ->
    let nonErrorVariants = filter (not . isErrorVariant) variants
    in length nonErrorVariants > 1  -- Multiple variants = streaming
  _ -> False
```

**Problem**: This breaks for mega-unions like `ArborEvent` (30 variants shared across all methods). The heuristic assumes all arbor methods are streaming.

**Solution**: Add explicit `streaming: bool` to server-side `MethodSchema`, set from Rust `impl Stream<Item=T>` detection.

### 2. Discriminator Detection

The IR assumes all enums use `"type"` as discriminator:

```haskell
inferDiscriminator :: [VariantDef] -> Text
inferDiscriminator _ = "type"  -- Always "type"
```

**Problem**: Serde supports custom tag names (`#[serde(tag = "kind")]`), but we don't extract them.

**Impact**: Low - convention is to always use `"type"`.

### 3. Per-Method Return Types

Shared mega-unions like `ArborEvent` cause bloated generated types:

```typescript
// Generated: every arbor method returns all 30 variants
type TreeGetResult = ArborEvent;  // Could be just TreeData | Error
```

**Solution**: Refactor to per-method result types in Rust:
```rust
// Instead of ArborEvent for all methods:
fn tree_get(...) -> TreeGetResult    // TreeData | Error
fn tree_create(...) -> TreeCreateResult  // TreeCreated | Error
```

## Type Flow Diagram

```
           Rust                    JSON Schema                  IR                   TypeScript
    ┌────────────────┐         ┌────────────────┐        ┌────────────┐         ┌────────────────┐
    │ struct Position│  ─────> │ $defs.Position │  ───>  │ TypeDef    │  ─────> │ interface      │
    │   tree_id: Uuid│         │   properties:  │        │   KindStruct│        │   Position {   │
    │   node_id: Uuid│         │     tree_id... │        │   fields... │        │     tree_id:   │
    └────────────────┘         └────────────────┘        └────────────┘         └────────────────┘
                                                                                        │
    ┌────────────────┐         ┌────────────────┐        ┌────────────┐         ┌───────┴────────┐
    │ enum ChatEvent │  ─────> │ oneOf: [       │  ───>  │ TypeDef    │  ─────> │ type ChatEvent │
    │   Start{...}   │         │   {type:start} │        │   KindEnum │        │   = Start      │
    │   Content{...} │         │   {type:content│        │   variants │        │   | Content    │
    │   Complete{...}│         │  ]             │        │   ...      │        │   | Complete   │
    └────────────────┘         └────────────────┘        └────────────┘         └────────────────┘
```

## Architecture Design Principles

### 1. Schema as Membrane

The JSON Schema boundary decouples Rust internals from client generation. Changes to substrate types automatically flow through without synapse changes.

### 2. IR as Universal Target

The IR is language-agnostic - additional transpilers (Python, Swift, Go) consume the same IR format.

### 3. Streaming by Default

All methods return streams. "Non-streaming" methods are streams that emit exactly one event. This uniformity simplifies protocol handling.

### 4. Native Feel

Generated clients mirror hub structure:
```typescript
hub.cone.chat({ ... })    // Not hub.call("cone.chat", {...})
```

## Related Documents

- `16679613932789736703_compiler-architecture.md` - Detailed transpiler design
- `16680785784679922687_client-codegen-requirements.md` - Requirements analysis
- `16680606527890548735_schema-to-cli-pipeline.md` - CLI generation (synapse --help)
- `16680807091363337727_introspective-rpc-protocol.md` - RPC protocol details
