# Structured Parameter Representation

**Status**: Proposal
**Date**: 2026-01-04

## Problem Statement

Currently, substrate's wire format for method schemas uses raw JSON Schema:

```rust
// Current: hub-core/src/plexus/schema.rs
pub struct MethodSchema {
    pub name: String,
    pub description: String,
    pub hash: String,
    pub params: Option<schemars::Schema>,   // Raw JSON Schema
    pub returns: Option<schemars::Schema>,  // Raw JSON Schema
    pub streaming: bool,
}
```

This creates problems:

1. **Duplicated parsing logic**: Every client (Synapse, TypeScript SDK, future clients) must independently parse JSON Schema and extract structure
2. **Inconsistent handling**: Complex patterns handled differently by each client
3. **Lost information**: When clients can't parse a pattern, they fall back to "any" type, losing type safety
4. **No single source of truth**: Schema interpretation happens N times in N clients instead of once at the source

### Current Flow

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Rust Types    │     │  JSON Schema    │     │  Client Types   │
│   (source)      │────►│  (wire format)  │────►│  (target)       │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │                       │                       │
   schemars                 raw JSON               N parsers
   (one place)              Schema             (duplicated work)
```

Every client must:
- Parse `oneOf` with `const` discriminators to detect tagged unions
- Resolve `$ref` to find type definitions
- Detect `anyOf` patterns for optional references
- Handle `additionalProperties` for maps
- Recognize format hints like `uuid`, `int64`

This is the **same work done N times**.

## Current State: Synapse IR Analysis

The Synapse IR Builder (`Synapse.IR.Builder`) already implements a "middle ground" approach - parsing JSON Schema into structured types, with fallback to `RefAny` for complex cases.

### Patterns Synapse Handles Well

| Pattern | JSON Schema | IR Type |
|---------|------------|---------|
| Primitives | `{"type": "string"}` | `RefPrimitive "string" Nothing` |
| Formatted | `{"type": "string", "format": "uuid"}` | `RefPrimitive "string" (Just "uuid")` |
| Arrays | `{"type": "array", "items": {...}}` | `RefArray (inner type)` |
| Optional | `{"type": ["string", "null"]}` | `RefOptional (RefPrimitive "string")` |
| Named refs | `{"$ref": "#/$defs/Foo"}` | `RefNamed "Foo"` |
| Optional refs | `{"anyOf": [{"$ref": "..."}, {"type": "null"}]}` | `RefOptional (RefNamed "Foo")` |
| Tagged unions | `{"oneOf": [{...}, {...}]}` with `const` discriminator | `KindEnum "type" [variants]` |
| String enums | `{"enum": ["a", "b", "c"]}` | `KindStringEnum ["a", "b", "c"]` |
| Structs | `{"type": "object", "properties": {...}}` | `KindStruct [fields]` |

### Patterns That Fall to RefAny

| Pattern | Example | Why |
|---------|---------|-----|
| Multiple refs in anyOf | `anyOf: [$ref A, $ref B]` | Ambiguous without discriminator |
| Untagged unions | `{"untagged": true}` enums | No discriminator in wire format |
| Complex nested | `Result<Option<Vec<T>>, E>` | Deep composition |
| additionalProperties | `HashMap<String, T>` | Map type not yet modeled |

When a pattern can't be parsed, Synapse falls back to `RefAny`, which means:
- CLI can't provide typed help or completion
- Code generators emit `any` / `unknown` types
- Type safety is lost for that parameter

### The Key Insight

Synapse already does this work. The question is: **should substrate emit structured types directly instead of raw JSON Schema, so all clients benefit?**

## Proposed Solution: Structured with Escape Hatch

Move the schema interpretation from clients to the source (hub-macro), emitting a structured representation with an escape hatch for complex cases.

### Core Types

```rust
// In hub-core/src/plexus/schema.rs

pub struct MethodSchema {
    pub name: String,
    pub description: String,
    pub hash: String,
    pub params: Vec<ParamDef>,              // Structured params
    pub types: HashMap<String, TypeDef>,    // Hoisted $defs
    pub returns: Option<ReturnDef>,
    pub streaming: bool,
}

pub struct ParamDef {
    pub name: String,
    pub param_type: ParamType,
    pub required: bool,
    pub description: Option<String>,
    pub default: Option<serde_json::Value>,
}

pub enum ParamType {
    // Structured (common cases - ~95% of params)
    Primitive { name: String, format: Option<String> },
    Ref(String),                    // Reference to types map
    Array(Box<ParamType>),
    Optional(Box<ParamType>),

    // Escape hatch (complex cases)
    Raw(serde_json::Value),         // Fall back to JSON Schema fragment
}

pub struct TypeDef {
    pub name: String,
    pub description: Option<String>,
    pub kind: TypeKind,
}

pub enum TypeKind {
    // Structured
    Struct { fields: Vec<ParamDef> },
    TaggedUnion {
        tagging: Tagging,
        variants: Vec<VariantDef>
    },
    StringEnum { values: Vec<String> },
    Alias(ParamType),

    // Escape hatch
    Raw(serde_json::Value),
}

pub enum Tagging {
    Internal { discriminator: String },  // #[serde(tag = "type")]
    External,                             // Default Rust enum
    Adjacent { tag: String, content: String },
}

pub struct VariantDef {
    pub name: String,
    pub description: Option<String>,
    pub payload: VariantPayload,
}

pub enum VariantPayload {
    Unit,
    Newtype(ParamType),
    Struct { fields: Vec<ParamDef> },
}

pub struct ReturnDef {
    pub return_type: ParamType,
    // For streaming methods, additional info about terminal variants
    pub terminal_variants: Option<Vec<String>>,
}
```

### Transformation Diagram

```
                    COMPILE TIME                           WIRE FORMAT
                    (hub-macro)                           (to clients)

┌─────────────────────────────────────────────────────────────────────────┐
│                                                                         │
│   Rust Types                    schemars                Structured      │
│   with serde                    Schema                  Schema          │
│   attributes                    (JSON)                  (new format)    │
│                                                                         │
│   ┌─────────────┐            ┌─────────────┐         ┌─────────────┐   │
│   │ #[serde(    │            │ {           │         │ ParamType:: │   │
│   │   tag="type"│   schemars │   "oneOf":  │ pattern │ Ref("Cone   │   │
│   │ )]          │──────────►│     [...]   │ detect  │ Identifier")│   │
│   │ enum Cone   │            │ }           │────────►│             │   │
│   │ Identifier  │            │             │         │ TypeKind::  │   │
│   │ {...}       │            └─────────────┘         │ TaggedUnion │   │
│   └─────────────┘                  │                 └─────────────┘   │
│                                    │                        │          │
│                                    │                        │          │
│                         ┌──────────┴──────────┐            │          │
│                         │  Pattern not         │            │          │
│                         │  recognized?         │            │          │
│                         └──────────┬──────────┘            │          │
│                                    │                        │          │
│                                    ▼                        │          │
│                         ┌─────────────────────┐            │          │
│                         │ ParamType::Raw(     │            │          │
│                         │   json_schema_frag  │◄───────────┘          │
│                         │ )                   │  escape hatch         │
│                         └─────────────────────┘                       │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
                                      │
                                      │ Wire (SSE/HTTP)
                                      ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                             CLIENTS                                     │
│                                                                         │
│   ┌───────────────┐   ┌───────────────┐   ┌───────────────┐           │
│   │    Synapse    │   │  TypeScript   │   │    Future     │           │
│   │   (Haskell)   │   │     SDK       │   │   Clients     │           │
│   └───────┬───────┘   └───────┬───────┘   └───────┬───────┘           │
│           │                   │                   │                    │
│           ▼                   ▼                   ▼                    │
│   ┌───────────────┐   ┌───────────────┐   ┌───────────────┐           │
│   │ Trivial       │   │ Direct TS     │   │ Native types  │           │
│   │ unfold to IR  │   │ type mapping  │   │ (no parsing)  │           │
│   └───────────────┘   └───────────────┘   └───────────────┘           │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
```

### When to Use Structured vs Raw

| Scenario | Use |
|----------|-----|
| Simple primitives | `ParamType::Primitive` |
| Named type reference | `ParamType::Ref` |
| `Option<T>` | `ParamType::Optional(inner)` |
| `Vec<T>` | `ParamType::Array(inner)` |
| Struct with fields | `TypeKind::Struct` |
| `#[serde(tag = "...")]` enum | `TypeKind::TaggedUnion` |
| Simple string enum | `TypeKind::StringEnum` |
| `serde_json::Value` | `ParamType::Raw({})` (intentionally dynamic) |
| Unrecognized pattern | `ParamType::Raw(schema_fragment)` (escape hatch) |
| `HashMap<String, T>` | `TypeKind::Raw` (until map support added) |

## Wire Format Examples

### Example 1: Simple Method (`echo.once`)

**Rust source:**
```rust
pub fn once(message: String) -> String { ... }
```

**Current wire format (raw JSON Schema):**
```json
{
  "name": "once",
  "description": "Echo a simple message once",
  "params": {
    "type": "object",
    "properties": {
      "message": { "type": "string", "description": "The message to echo" }
    },
    "required": ["message"]
  }
}
```

**Proposed wire format (structured):**
```json
{
  "name": "once",
  "description": "Echo a simple message once",
  "params": [
    {
      "name": "message",
      "param_type": { "Primitive": { "name": "string", "format": null } },
      "required": true,
      "description": "The message to echo"
    }
  ],
  "types": {},
  "returns": { "return_type": { "Primitive": { "name": "string", "format": null } } }
}
```

### Example 2: Tagged Union (`cone.chat` with ConeIdentifier)

**Rust source:**
```rust
#[derive(JsonSchema)]
#[serde(tag = "type")]
pub enum ConeIdentifier {
    #[serde(rename = "by_name")]
    ByName { name: String },
    #[serde(rename = "by_id")]
    ById { id: Uuid },
}

pub fn chat(identifier: ConeIdentifier, prompt: String) -> impl Stream<Item = ChatEvent> { ... }
```

**Current wire format (raw JSON Schema):**
```json
{
  "name": "chat",
  "params": {
    "$defs": {
      "ConeIdentifier": {
        "oneOf": [
          {
            "type": "object",
            "properties": {
              "type": { "const": "by_name" },
              "name": { "type": "string" }
            },
            "required": ["type", "name"]
          },
          {
            "type": "object",
            "properties": {
              "type": { "const": "by_id" },
              "id": { "type": "string", "format": "uuid" }
            },
            "required": ["type", "id"]
          }
        ]
      }
    },
    "properties": {
      "identifier": { "$ref": "#/$defs/ConeIdentifier" },
      "prompt": { "type": "string" }
    },
    "required": ["identifier", "prompt"]
  }
}
```

**Proposed wire format (structured):**
```json
{
  "name": "chat",
  "params": [
    {
      "name": "identifier",
      "param_type": { "Ref": "ConeIdentifier" },
      "required": true
    },
    {
      "name": "prompt",
      "param_type": { "Primitive": { "name": "string", "format": null } },
      "required": true
    }
  ],
  "types": {
    "ConeIdentifier": {
      "name": "ConeIdentifier",
      "kind": {
        "TaggedUnion": {
          "tagging": { "Internal": { "discriminator": "type" } },
          "variants": [
            {
              "name": "by_name",
              "payload": {
                "Struct": {
                  "fields": [
                    { "name": "name", "param_type": { "Primitive": { "name": "string" } }, "required": true }
                  ]
                }
              }
            },
            {
              "name": "by_id",
              "payload": {
                "Struct": {
                  "fields": [
                    { "name": "id", "param_type": { "Primitive": { "name": "string", "format": "uuid" } }, "required": true }
                  ]
                }
              }
            }
          ]
        }
      }
    }
  }
}
```

### Example 3: Complex Case Falling to Raw

**Rust source:**
```rust
// Nested generics that hub-macro can't fully analyze
pub fn complex(input: Result<Option<Vec<Foo>>, BarError>) { ... }
```

**Proposed wire format (escape hatch):**
```json
{
  "name": "complex",
  "params": [
    {
      "name": "input",
      "param_type": {
        "Raw": {
          "oneOf": [
            {
              "type": "object",
              "properties": {
                "Ok": {
                  "anyOf": [
                    { "type": "array", "items": { "$ref": "#/$defs/Foo" } },
                    { "type": "null" }
                  ]
                }
              }
            },
            {
              "type": "object",
              "properties": {
                "Err": { "$ref": "#/$defs/BarError" }
              }
            }
          ]
        }
      },
      "required": true
    }
  ],
  "types": {
    "Foo": { "name": "Foo", "kind": { "Raw": { "type": "object", "..." } } },
    "BarError": { "name": "BarError", "kind": { "Raw": { "..." } } }
  }
}
```

The client knows this is a complex type (`Raw`) and can:
1. Accept it as JSON in CLI
2. Generate `any` type in code
3. Display the raw schema in help

## Implementation in hub-macro

The transformation happens at compile time in hub-macro, processing schemars output:

```rust
// hub-macro/src/codegen/structured_schema.rs

/// Transform schemars Schema to structured representation
fn schema_to_param_type(schema: &schemars::Schema, defs: &HashMap<String, schemars::Schema>) -> ParamType {
    match classify_schema(schema) {
        SchemaClass::Primitive(name, format) =>
            ParamType::Primitive { name, format },

        SchemaClass::Ref(name) =>
            ParamType::Ref(name),

        SchemaClass::Array(item_schema) =>
            ParamType::Array(Box::new(schema_to_param_type(item_schema, defs))),

        SchemaClass::Optional(inner) =>
            ParamType::Optional(Box::new(schema_to_param_type(inner, defs))),

        SchemaClass::TaggedUnion { discriminator, variants } =>
            // Process inline or add to types map
            ...,

        SchemaClass::Unknown =>
            // Escape hatch: emit raw JSON Schema
            ParamType::Raw(serde_json::to_value(schema).unwrap()),
    }
}

/// Classify a JSON Schema fragment
fn classify_schema(schema: &schemars::Schema) -> SchemaClass {
    // Check for $ref
    if let Some(ref_path) = schema.reference() {
        return SchemaClass::Ref(extract_ref_name(ref_path));
    }

    // Check for array
    if schema.is_array() {
        return SchemaClass::Array(schema.items().unwrap());
    }

    // Check for nullable (type: ["string", "null"])
    if let Some(inner) = detect_optional(schema) {
        return SchemaClass::Optional(inner);
    }

    // Check for oneOf with const discriminator (tagged union)
    if let Some(union_info) = detect_tagged_union(schema) {
        return SchemaClass::TaggedUnion(union_info);
    }

    // Check for primitive
    if let Some((name, format)) = detect_primitive(schema) {
        return SchemaClass::Primitive(name, format);
    }

    // Not recognized - escape hatch
    SchemaClass::Unknown
}
```

### Detection Heuristics

**Tagged Union Detection:**
```rust
fn detect_tagged_union(schema: &schemars::Schema) -> Option<TaggedUnionInfo> {
    let one_of = schema.one_of()?;

    // Find the discriminator field (field with const value in all variants)
    let discriminator = find_common_const_field(one_of)?;

    // Extract variants
    let variants = one_of.iter()
        .map(|variant| {
            let tag_value = get_const_value(variant, &discriminator)?;
            let fields = extract_variant_fields(variant, &discriminator);
            Some(VariantInfo { name: tag_value, fields })
        })
        .collect::<Option<Vec<_>>>()?;

    Some(TaggedUnionInfo { discriminator, variants })
}
```

**Optional Reference Detection:**
```rust
fn detect_optional_ref(schema: &schemars::Schema) -> Option<String> {
    let any_of = schema.any_of()?;
    if any_of.len() != 2 { return None; }

    // One must be null, one must be $ref
    let null_variant = any_of.iter().find(|s| s.is_null())?;
    let ref_variant = any_of.iter().find(|s| s.reference().is_some())?;

    ref_variant.reference().map(extract_ref_name)
}
```

## Client Impact

### Synapse (Haskell)

Current IR Builder parses JSON Schema into `TypeRef` / `TypeKind`. With structured params, this becomes a trivial unfold:

```haskell
-- Before: complex JSON Schema parsing
schemaToTypeRef :: Value -> TypeRef
schemaToTypeRef (Object o)
  | Just (String ref) <- KM.lookup "$ref" o = RefNamed (extractRefName ref)
  | Just (String "array") <- KM.lookup "type" o = ...
  | Just (Array types) <- KM.lookup "type" o = ...  -- nullable detection
  | Just (Array options) <- KM.lookup "anyOf" o = ... -- optional ref detection
  | ...

-- After: trivial case match
paramTypeToTypeRef :: ParamType -> TypeRef
paramTypeToTypeRef = \case
  Primitive name format -> RefPrimitive name format
  Ref name -> RefNamed name
  Array inner -> RefArray (paramTypeToTypeRef inner)
  Optional inner -> RefOptional (paramTypeToTypeRef inner)
  Raw _ -> RefAny  -- escape hatch preserves fallback behavior
```

**ParamF from unified-cli-tree.md becomes trivial:**

```haskell
-- Unfold structured params directly into ParamF
paramsToParamTree :: [ParamDef] -> Map Text TypeDef -> [Fix ParamF]
paramsToParamTree params types = map (unfoldParam types) params

unfoldParam :: Map Text TypeDef -> ParamDef -> Fix ParamF
unfoldParam types pd = Fix $ case pdType pd of
  Primitive name fmt -> FieldNode (pdName pd) (toPrimType name fmt)

  Ref typeName -> case Map.lookup typeName types of
    Just td -> case tdKind td of
      TaggedUnion _ variants ->
        ParamNode (pdName pd) (PKTaggedUnion (map vdName variants)) (Just variantNodes)
      Struct fields ->
        ParamNode (pdName pd) (PKObject []) Nothing  -- flatten or expand
      StringEnum values ->
        ParamNode (pdName pd) (PKStringEnum values) Nothing
    Nothing -> FieldNode (pdName pd) PTAny  -- undefined ref

  Array inner ->
    ParamNode (pdName pd) (PKArray (toParamKind inner)) Nothing

  Optional inner ->
    -- Mark as optional, recurse
    ...

  Raw _ ->
    FieldNode (pdName pd) PTAny  -- escape hatch = any
```

### TypeScript SDK (hub-codegen)

Direct mapping to TypeScript types:

```typescript
// Before: parse JSON Schema, detect patterns
function schemaToTsType(schema: JsonSchema): TsType { ... }

// After: pattern match on ParamType
function paramTypeToTsType(pt: ParamType): TsType {
  switch (pt.type) {
    case 'Primitive': return primitiveToTs(pt.name, pt.format);
    case 'Ref': return { kind: 'reference', name: pt.name };
    case 'Array': return { kind: 'array', element: paramTypeToTsType(pt.inner) };
    case 'Optional': return { kind: 'optional', inner: paramTypeToTsType(pt.inner) };
    case 'Raw': return { kind: 'any' };  // escape hatch
  }
}

function typeDefToTsType(td: TypeDef): TsDeclaration {
  switch (td.kind.type) {
    case 'TaggedUnion':
      return generateDiscriminatedUnion(td.name, td.kind.variants, td.kind.tagging);
    case 'Struct':
      return generateInterface(td.name, td.kind.fields);
    case 'StringEnum':
      return generateStringLiteralUnion(td.name, td.kind.values);
    case 'Raw':
      return { name: td.name, type: 'unknown' };
  }
}
```

### Future Clients

Any new client (Swift SDK, Python SDK, etc.) benefits immediately:
- No JSON Schema parsing logic to implement
- Structured types map directly to language constructs
- Escape hatch provides graceful degradation

## Migration Path

### Phase 1: Add Structured Fields (Backwards Compatible)

```rust
pub struct MethodSchema {
    pub name: String,
    pub description: String,
    pub hash: String,

    // Existing (kept for backwards compat)
    #[serde(skip_serializing_if = "Option::is_none")]
    pub params: Option<schemars::Schema>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub returns: Option<schemars::Schema>,

    // New structured fields
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
    pub structured_params: Vec<ParamDef>,
    #[serde(default, skip_serializing_if = "HashMap::is_empty")]
    pub types: HashMap<String, TypeDef>,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub structured_returns: Option<ReturnDef>,

    pub streaming: bool,
}
```

Both representations emitted. Clients can migrate at their own pace.

### Phase 2: Client Adoption

- Synapse: Use `structured_params` when present, fall back to `params`
- TypeScript SDK: Same pattern
- Other clients: Implement against structured representation

### Phase 3: Deprecate Raw Fields

Once all clients have migrated:
- Mark `params` and `returns` as deprecated
- Eventually remove (major version bump)

## Relationship to Other Documents

### unified-cli-tree.md

The `ParamF` functor proposed there becomes a trivial unfold of the structured representation:

```
MethodSchema.structured_params  ──unfold──►  [Fix ParamF]
                │                                  │
                │                                  │
                ▼                                  ▼
    Already has structure              CLI navigation tree
    (TaggedUnion, Struct, etc.)        (help, completion, parsing)
```

The key insight: **ParamF is the CLI-specific view of the same structure**. With structured params, building ParamF is a simple transformation rather than JSON Schema interpretation.

### method-schema-spec.md

That document defines the JSON Schema patterns we support. This proposal adds:
- **Detection heuristics**: How to recognize each pattern
- **Structured output**: What to emit for each pattern
- **Escape hatch**: What to do when pattern isn't recognized

The supported patterns remain the same; we're just moving interpretation to the source.

### ir-codegen-chain.md

The IR becomes simpler:
```
Schema (raw)          →  IR (via Builder)       →  Codegen
        ▲                       │
        │                       │
        │    OR (proposed)      ▼
        │
Schema (structured)   →  IR (trivial transform) →  Codegen
```

## Open Questions

1. **Should `Raw` include original schemars::Schema or JSON Value?**
   - Schema: Clients can use schemars-compatible parsers
   - Value: More portable, no schemars dependency in clients

2. **How deep should type hoisting go?**
   - All types in `types` map (flat)
   - Only top-level $defs (current behavior)
   - Inline simple types, hoist complex ones

3. **Should we version the structured format?**
   - Add `schema_version: "1.0"` field
   - Allows evolution without breaking clients

4. **What about additionalProperties (maps)?**
   - Add `TypeKind::Map { value_type: ParamType }`
   - Or fall to `Raw` until v2

## Summary

| Aspect | Current | Proposed |
|--------|---------|----------|
| Wire format | Raw JSON Schema | Structured types + escape hatch |
| Parsing location | Every client | Once at source (hub-macro) |
| Client complexity | High (JSON Schema interpretation) | Low (pattern match) |
| Type safety | Lost on complex patterns | Preserved or explicitly `Raw` |
| Backwards compat | N/A | Phase 1 adds fields, Phase 3 removes old |

The "structured with escape hatch" approach gives us:
- **95% case**: Full type information, trivial client consumption
- **5% edge cases**: Graceful fallback to raw schema, no worse than today
- **Clear signal**: `Raw` indicates "hub-macro couldn't classify this"
