# Schema as Membrane: The Bidirectional Lens Pattern

## Observation

While building the dynamic CLI (synapse) and the MCP bridge for Plexus, a pattern emerged: the same schema serves multiple purposes that initially appear distinct but are actually projections of the same structure.

## The Pattern

A schema defines three operations that form a complete interface:

```
                    ┌─────────────────┐
                    │     Schema      │
                    │   (TypeInfo)    │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼              │              ▼
        describe()           │         parse()
              │              │              │
              ▼              │              ▼
      Help/Documentation     │      Input Handling
                             │
                             ▼
                         render()
                             │
                             ▼
                      Output Display
```

### Concrete Example: `ConeIdentifier`

```rust
#[derive(JsonSchema)]
pub enum ConeIdentifier {
    ByName { name: String },
    ById { id: Uuid },
}
```

This single type definition generates:

| Operation | Output |
|-----------|--------|
| `describe()` | `--id TEXT  Cone name or UUID (e.g., 'my-assistant')` |
| `parse("haiku-test")` | `ConeIdentifier::ByName { name: "haiku-test" }` |
| `render(cone_list)` | Formatted table of cones |

## The Insight

**A membrane is a bidirectional lens between structured types and human-readable strings.**

```haskell
class Membrane m where
  describe :: m -> Text           -- What receptors are exposed?
  parse    :: m -> Text -> Value  -- Import through membrane
  render   :: m -> Value -> Text  -- Export through membrane
```

The CLI isn't a separate abstraction - it **falls out** from having this membrane:

- `describe()` → generates help text and usage
- `parse()` → handles user input
- `render()` → formats command output

## Evidence in Synapse

Synapse demonstrates this pattern:

1. **Schema fetching**: `GET /rpc/schema` returns `ActivationFullSchema`
2. **CLI generation**: Schema → optparse-applicative parser (dynamic, at runtime)
3. **Help rendering**: Same schema → `--help` output
4. **Output rendering**: Return types → mustache templates → terminal

The entire CLI is generated from schemas. No hand-written argument parsing.

## Generalization

If membranes are projections of schemas, then different membrane types are just different rendering targets:

| Membrane Type | describe() | parse() | render() |
|--------------|------------|---------|----------|
| CLI | `--help` flags | argv parsing | terminal text |
| Web UI | form fields | form submission | HTML/React |
| Chat | natural language | NLU/intent | natural language |
| MCP | tool schemas | JSON params | tool results |

The same `ActivationFullSchema` could theoretically generate all of these membranes.

## Implications

### 1. Single Source of Truth
The function signature IS the schema IS the interface. No duplication.

```rust
#[hub_method(params(
    identifier = "Cone name or UUID"
))]
async fn get(&self, identifier: ConeIdentifier) -> impl Stream<Item = ConeEvent>
```

This one definition provides:
- Type safety (Rust compiler)
- API schema (schemars)
- CLI interface (synapse)
- MCP tools (rmcp bridge)
- Documentation (--help)

### 2. Parser-Printer Duality
Every schema is implicitly a parser-printer pair. The description that tells humans how to format input is the same structure that parses that input.

### 3. Membrane Composition
If membranes are schema projections, then composing membranes = composing schemas. Plexus already does this - multiple activations compose into one unified membrane.

## Open Questions

1. **Natural Language Interface**: Can the same schema drive a conversational interface? The `describe()` output is already human-readable...

2. **Bidirectional Sync**: If the schema changes, all interfaces update automatically. But what about the reverse - can interface feedback inform schema evolution?

   **Note (Dec 2024)**: This is interesting but out of scope. Currently, the human developer is the channel through which interface feedback affects the schema. The workflow is: notice something missing/awkward in the interface → modify the backend and schema. Automating this would require something like a "submit your requests here" button with an agent requesting human approval for implementation changes. We're not building that now, but the pattern is worth noting: interface evolution currently flows through human judgment.

3. **Error Messages**: The lens pattern suggests error messages should also derive from the schema. "Expected Cone name or UUID" comes from the same place as the help text.

## Refactoring Synapse to the Schema-as-Membrane Model

The goal is to represent the existing synapse code in terms of the `Membrane<T>` abstraction without losing or gaining functionality. This is a pure restructuring to make the pattern explicit.

### Current Architecture (Implicit)

```
┌─────────────────────────────────────────────────────────────────┐
│                     Current Synapse                              │
├─────────────────────────────────────────────────────────────────┤
│  Schema Fetch          CLI Generation        Output Rendering   │
│  ─────────────         ──────────────        ────────────────   │
│  GET /rpc/schema  →    optparse-applicative  mustache templates │
│                        (buildParser)         (TemplateResolver) │
│                                                                 │
│  These are separate concerns that happen to share schema data   │
└─────────────────────────────────────────────────────────────────┘
```

### Target Architecture (Explicit)

```
┌─────────────────────────────────────────────────────────────────┐
│                  Schema-as-Membrane Synapse                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Membrane                                                       │
│  ├── describe :: Schema -> HelpText      (receptors exposed)   │
│  ├── parse    :: Schema -> String -> Either Error Params       │
│  └── render   :: Schema -> Value -> Text (export to outside)   │
│                                                                 │
│  CLI = Membrane applied to ActivationFullSchema                 │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

### Mapping Existing Code to the Abstraction

| Abstraction | Current Implementation | Location |
|-------------|----------------------|----------|
| `describe()` | `buildActivationParser` + help generation | `Dynamic.hs` |
| `parse()` | `buildMethodParser` + param extraction | `Dynamic.hs` |
| `render()` | `TemplateResolver` + handlebars | `Renderer.hs` |
| `Schema` | `ActivationFullSchema` / `MethodSchemaInfo` | fetched from substrate |

## Membrane as a Separate Haskell Library

Membrane is implemented in **Haskell**, not Rust. The architecture:

```
┌─────────────────────┐         JSON          ┌─────────────────────┐
│      Substrate      │ ──────────────────→   │      Membrane       │
│       (Rust)        │   ActivationSchema    │      (Haskell)      │
│                     │                       │                     │
│  - Activations      │                       │  - CLI generation   │
│  - Plexus           │                       │  - MCP bridge       │
│  - schemars         │                       │  - Output rendering │
└─────────────────────┘                       └─────────────────────┘
```

### The Backbone Type (Contract)

This is THE contract between Substrate and Membrane. Substrate produces it as JSON, Membrane consumes it. No other coupling exists.

```haskell
-- membrane/src/Membrane/Schema.hs
-- THE CONTRACT: Substrate produces this, Membrane consumes it

module Membrane.Schema where

import Data.Aeson (FromJSON, ToJSON, Value)
import Data.Text (Text)
import GHC.Generics (Generic)

-- | Complete description of an activation's surface area
data ActivationSchema = ActivationSchema
  { asNamespace   :: !Text
  , asVersion     :: !Text
  , asDescription :: !Text
  , asMethods     :: ![MethodSchema]
  } deriving (Eq, Show, Generic)
    deriving anyclass (FromJSON, ToJSON)

-- | Everything needed to describe/parse/render a single method
data MethodSchema = MethodSchema
  { msName        :: !Text
  , msDescription :: !Text
  , msParams      :: ![ParamSchema]
  , msReturns     :: !(Maybe TypeSchema)
  } deriving (Eq, Show, Generic)
    deriving anyclass (FromJSON, ToJSON)

-- | A single parameter - explicit required/default for CLI generation
data ParamSchema = ParamSchema
  { psName        :: !Text
  , psDescription :: !Text
  , psType        :: !TypeSchema
  , psRequired    :: !Bool
  , psDefault     :: !(Maybe Value)
  } deriving (Eq, Show, Generic)
    deriving anyclass (FromJSON, ToJSON)

-- | Self-contained type info - no $ref resolution at membrane layer
data TypeSchema
  = TString   { tsFormat :: !(Maybe StringFormat), tsEnum :: !(Maybe [Text]) }
  | TNumber   { tsMin :: !(Maybe Double), tsMax :: !(Maybe Double) }
  | TInteger  { tsMinInt :: !(Maybe Int), tsMaxInt :: !(Maybe Int) }
  | TBoolean
  | TArray    { tsItems :: !TypeSchema }
  | TObject   { tsProperties :: ![ParamSchema], tsAdditional :: !Bool }
  | TUnion    { tsVariants :: ![TypeSchema] }
  | TAny
  deriving (Eq, Show, Generic)
  deriving anyclass (FromJSON, ToJSON)

data StringFormat = FmtUuid | FmtDateTime | FmtUri | FmtEmail
  deriving (Eq, Show, Generic)
  deriving anyclass (FromJSON, ToJSON)
```

### The Membrane Typeclass

```haskell
-- membrane/src/Membrane/Class.hs

module Membrane.Class where

import Data.Aeson (Value)
import Data.Text (Text)
import Membrane.Schema

-- | A bidirectional membrane over a schema
-- Like a cell membrane: controls what passes through and how it's translated
class Membrane m where
  -- | The schema this membrane wraps
  type Schema m

  -- | What receptors are exposed? (help text, usage)
  describe :: m -> Text

  -- | Import through membrane: human input -> structured params
  parse :: m -> Text -> Either ParseError Value

  -- | Export through membrane: structured output -> human-readable
  render :: m -> Value -> Text

data ParseError = ParseError
  { peMessage  :: Text
  , peExpected :: Maybe Text
  , peGot      :: Maybe Text
  } deriving (Eq, Show)
```

### Concrete Membrane Instances

```haskell
-- membrane/src/Membrane/CLI.hs
-- CLI membrane - generates optparse-applicative parsers

module Membrane.CLI where

import Membrane.Class
import Membrane.Schema
import Options.Applicative (Parser)

newtype CLIMembrane = CLIMembrane MethodSchema

instance Membrane CLIMembrane where
  type Schema CLIMembrane = MethodSchema

  describe (CLIMembrane m) = buildHelpText m
  parse (CLIMembrane m) input = runParser (buildParser m) input
  render (CLIMembrane m) val = renderWithTemplate m val

-- | Build an optparse-applicative parser from schema
buildParser :: MethodSchema -> Parser Value
buildParser = ...

-- membrane/src/Membrane/MCP.hs
-- MCP membrane - generates MCP tool schemas

module Membrane.MCP where

newtype MCPMembrane = MCPMembrane MethodSchema

instance Membrane MCPMembrane where
  type Schema MCPMembrane = MethodSchema

  describe (MCPMembrane m) = buildToolSchema m
  parse (MCPMembrane m) input = parseJsonParams m input
  render (MCPMembrane m) val = toToolResult val
```

### Why Haskell?

| Capability | Why it matters for Membrane |
|------------|----------------------------|
| **Type families** | `type Schema m` associates schema type with membrane |
| **GADTs** | Type-safe schema representation with constraints |
| **Deriving via** | Reduce boilerplate for similar membranes |
| **optparse-applicative** | Already a perfect fit for `describe` + `parse` |
| **Aeson** | Seamless JSON ↔ Haskell for the backbone type |
| **Profunctors** | Parser-printer duality is literally a profunctor |
| **Free monads** | Compose membrane operations as interpreters |

### Substrate's Responsibility

Substrate (Rust) must produce `ActivationSchema` as JSON. This happens in Plexus:

```rust
// substrate/src/plexus/schema.rs
// Convert internal types to the Membrane contract

impl Plexus {
    pub fn export_membrane_schema(&self) -> Vec<membrane::ActivationSchema> {
        self.list_full_schemas()
            .into_iter()
            .map(|s| s.into())  // Convert to membrane format
            .collect()
    }
}

impl From<ActivationFullSchema> for membrane::ActivationSchema {
    fn from(s: ActivationFullSchema) -> Self {
        // Flatten schemars::Schema into TypeSchema
        // Resolve $refs
        // Extract required/default from JSON Schema
        ...
    }
}
```

### Refactoring Synapse

Synapse becomes a thin wrapper that:
1. Fetches `ActivationSchema` from Substrate
2. Wraps each method in `CLIMembrane`
3. Uses the Membrane library for all CLI logic

```haskell
-- synapse/src/Main.hs

main :: IO ()
main = do
  schemas <- fetchSchemas "ws://127.0.0.1:4444"
  let membranes = concatMap activationToMembranes schemas
  runCLI membranes

activationToMembranes :: ActivationSchema -> [CLIMembrane]
activationToMembranes act = CLIMembrane <$> asMethods act
```

### What Stays The Same

- All existing parsing logic (moves into Membrane library)
- All existing template resolution (moves into Membrane library)
- All existing help text generation (moves into Membrane library)
- RPC calling mechanics (stays in synapse)
- Schema fetching (stays in synapse)

### What Changes

- Logic extracted into reusable `membrane` library
- Synapse becomes a thin orchestrator
- Same Membrane library can power MCP, Web UI, etc.

## High-Level Implementation Vision

### Module Structure

```
membrane/                          synapse/
├── Membrane/                      ├── Main.hs          (entry point)
│   ├── Schema.hs     (backbone)   ├── Transport.hs     (WebSocket to Substrate)
│   ├── Class.hs      (typeclass)  └── App.hs           (thin orchestrator)
│   ├── CLI.hs        (CLIMembrane)
│   ├── MCP.hs        (MCPMembrane)
│   ├── Surface.hs    (combined)
│   └── Render.hs     (templates)
```

### The Main Entry Point

```haskell
-- synapse/Main.hs

module Main where

import Membrane
import Transport (connectSubstrate, fetchSchema)

main :: IO ()
main = do
  -- 1. Connect & fetch
  conn    <- connectSubstrate "ws://127.0.0.1:4444"
  schemas <- fetchSchema conn

  -- 2. Build membrane surface
  let surface = buildSurface schemas

  -- 3. Run CLI through the membrane
  runCLI conn surface
```

### The Surface (All Membranes Combined)

```haskell
-- membrane/Membrane/Surface.hs

-- | A Surface is a collection of membranes exposing an entire API
data Surface = Surface
  { surfaceActivations :: [ActivationSurface]
  }

data ActivationSurface = ActivationSurface
  { asNamespace :: Text
  , asMethods   :: [CLIMembrane]
  }

-- | Build a complete surface from schemas
buildSurface :: [ActivationSchema] -> Surface
buildSurface schemas = Surface
  { surfaceActivations = buildActivation <$> schemas
  }
  where
    buildActivation act = ActivationSurface
      { asNamespace = asNamespace act
      , asMethods   = CLIMembrane <$> asMethods act
      }
```

### Running the CLI

```haskell
-- membrane/Membrane/CLI.hs

-- | The entire CLI, derived from the surface
runCLI :: Connection -> Surface -> IO ()
runCLI conn surface = do
  let parser = surfaceToParser surface
  command <- execParser parser

  case command of
    Help          -> putStrLn (describeAll surface)
    Run ns method args -> do
      let membrane = lookupMembrane surface ns method
      case parse membrane args of
        Left err     -> die (renderParseError err)
        Right params -> do
          result <- call conn ns method params
          putStrLn (render membrane result)
```

### The Core Flow Visualized

```
┌─────────────────────────────────────────────────────────────────────────┐
│                              synapse                                     │
│                                                                          │
│   argv: ["cone", "chat", "--id", "haiku", "--prompt", "hello"]          │
│                              │                                           │
│                              ▼                                           │
│   ┌──────────────────────────────────────────────────────────────────┐  │
│   │                         Surface                                   │  │
│   │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐              │  │
│   │  │   arbor     │  │    cone     │  │    bash     │   ...        │  │
│   │  │  membranes  │  │  membranes  │  │  membranes  │              │  │
│   │  └─────────────┘  └──────┬──────┘  └─────────────┘              │  │
│   └──────────────────────────┼───────────────────────────────────────┘  │
│                              │                                           │
│                              ▼                                           │
│   ┌──────────────────────────────────────────────────────────────────┐  │
│   │                    CLIMembrane "chat"                             │  │
│   │                                                                   │  │
│   │   describe() → "--id TEXT  Cone name or UUID"                    │  │
│   │                "--prompt TEXT  Message to send"                  │  │
│   │                                                                   │  │
│   │   parse(argv) → Right { "identifier": {"by_name": "haiku"},     │  │
│   │                         "prompt": "hello" }                      │  │
│   │                                                                   │  │
│   │   render(result) → "Assistant: Hello! How can I help?"          │  │
│   └──────────────────────────────────────────────────────────────────┘  │
│                              │                                           │
│                              ▼                                           │
│                     call Substrate RPC                                   │
│                              │                                           │
│                              ▼                                           │
│                        stdout                                            │
└─────────────────────────────────────────────────────────────────────────┘
```

### The Membrane Doing Its Job

```haskell
-- What happens inside CLIMembrane

-- DESCRIBE: Schema → optparse-applicative help
describe (CLIMembrane m) = unlines
  [ msDescription m
  , ""
  , "Options:"
  , unlines [ "  --" <> psName p <> "  " <> psDescription p
            | p <- msParams m ]
  ]

-- PARSE: String args → JSON Value
parse (CLIMembrane m) args =
  let parser = foldr (<*>) (pure mempty) (paramToParser <$> msParams m)
  in execParserPure defaultPrefs parser (words args)

-- RENDER: JSON Value → Pretty terminal output
render (CLIMembrane m) val =
  case msReturns m of
    Nothing     -> encode val  -- fallback to JSON
    Just schema -> applyTemplate schema val
```

### The Dream: Same Schema, Multiple Membranes

```haskell
-- One schema, many surfaces

schema <- fetchSchema conn

-- CLI surface
let cli = buildSurface @CLIMembrane schema
runCLI cli

-- MCP surface (same schema!)
let mcp = buildSurface @MCPMembrane schema
serveMCP mcp

-- Web surface (same schema!)
let web = buildSurface @WebMembrane schema
serveWeb web
```

The **schema is fetched once**, then projected through different membranes to create different interfaces. The membrane IS the interface generator.

## Connection to Semantic Inflation

This pattern relates to the broader "semantic inflation" theme: turning vague specifications into concrete implementations. A schema is a compressed representation that "inflates" into:
- Type definitions
- Parsers
- Printers
- Documentation
- Membranes

The CLI membrane is just one inflation target among many.

## Implementation Requirements

This section documents what's needed to build membrane from scratch.

### Project Structure

```
membrane/
├── membrane.cabal           -- Package definition
├── src/
│   ├── Membrane.hs          -- Re-exports
│   └── Membrane/
│       ├── Schema.hs        -- Backbone types (THE contract)
│       ├── Class.hs         -- Membrane typeclass
│       ├── CLI.hs           -- CLIMembrane implementation
│       ├── MCP.hs           -- MCPMembrane implementation
│       ├── Surface.hs       -- Combined API surface
│       ├── Render.hs        -- Output rendering
│       ├── Render/
│       │   └── Template.hs  -- Template-based rendering
│       ├── Parse.hs         -- Input parsing
│       └── Parse/
│           └── Args.hs      -- CLI argument parsing
├── test/                    -- Test suite
└── app/
    └── Main.hs              -- Demo application
```

### Dependencies

```cabal
build-depends:
    base >= 4.14 && < 5,
    aeson >= 2.0,              -- JSON handling
    text >= 1.2,               -- Text type
    bytestring >= 0.10,        -- ByteString
    containers >= 0.6,         -- Map, Set
    unordered-containers,      -- HashMap
    vector >= 0.12,            -- Vector for arrays
    optparse-applicative,      -- CLI parser building
    mustache >= 2.4,           -- Template rendering
    mtl >= 2.2,                -- Monad transformers
    filepath,                  -- Path manipulation
    directory,                 -- File system operations
    scientific,                -- Numeric precision
    uuid,                      -- UUID handling
    time                       -- Timestamps
```

### GHC Extensions Required

```haskell
default-extensions:
    DerivingStrategies         -- Clean deriving
    DeriveGeneric              -- Generic instances
    DeriveAnyClass             -- FromJSON/ToJSON deriving
    OverloadedStrings          -- Text literals
    TypeFamilies               -- Associated types in Membrane class
    GADTs                      -- TypeSchema ADT
    DataKinds                  -- Type-level programming
    LambdaCase                 -- Pattern matching lambdas
    RecordWildCards            -- Field access shorthand
    NamedFieldPuns             -- Field punning
```

### Critical Implementation Details

#### 1. Schema Parsing from Substrate

Substrate (Rust) produces JSON via `schemars`. The membrane must:
- Handle `schemars` quirks (missing `type: "object"`, `$ref` pointers)
- Resolve all `$ref` references to inline schemas
- Convert JSON Schema → `TypeSchema` ADT

```haskell
-- In production, add schema normalization
normalizeSchema :: Value -> Either String ActivationSchema
normalizeSchema raw = do
  -- 1. Resolve all $refs
  resolved <- resolveRefs raw
  -- 2. Ensure type: "object" on all param schemas
  normalized <- ensureObjectTypes resolved
  -- 3. Parse into our types
  parseJSON normalized
```

#### 2. CLI Parser Generation

The `CLIMembrane` must generate `optparse-applicative` parsers dynamically:

```haskell
-- Key insight: build Parser from schema at runtime
methodToParser :: MethodSchema -> Parser Value
methodToParser m = Object . KM.fromList <$> traverse paramToParser (msParams m)

paramToParser :: ParamSchema -> Parser (Key, Value)
paramToParser p = (K.fromText (psName p),) <$> case psType p of
  TString _ _  -> String <$> strOption (long (T.unpack (psName p)) <> help ...)
  TInteger _ _ -> Number . fromIntegral <$> option auto (long ...)
  TBoolean     -> Bool <$> switch (long ...)
  ...
```

#### 3. Template Resolution

Templates follow a search path with fallbacks:

```haskell
templateSearchPaths :: IO [FilePath]
templateSearchPaths = do
  cwd <- getCurrentDirectory
  home <- getHomeDirectory
  pure
    [ cwd </> ".substrate" </> "templates"      -- Project-local
    , home </> ".config" </> "membrane" </> "templates"  -- User global
    -- Built-in templates embedded in binary
    ]
```

#### 4. Error Messages Derived from Schema

Parse errors should use schema information:

```haskell
buildParseError :: ParamSchema -> Text -> ParseError
buildParseError ParamSchema{..} got = ParseError
  { peMessage = "Invalid value for " <> psName
  , peExpected = Just $ describeType psType  -- "an integer between 0 and 100"
  , peGot = Just got
  , peLocation = Just psName
  }

describeType :: TypeSchema -> Text
describeType = \case
  TString _ (Just enum) -> "one of: " <> T.intercalate ", " enum
  TInteger (Just lo) (Just hi) -> "an integer between " <> show lo <> " and " <> show hi
  TBoolean -> "true or false"
  ...
```

### MVP Scope

Phase 1 (minimum viable membrane):
1. `Membrane.Schema` - Parse backbone types from JSON
2. `Membrane.Class` - Define typeclass
3. `Membrane.CLI` - Basic CLI generation (no templates)
4. Demo: hardcoded schema → working CLI

Phase 2:
1. Template-based rendering
2. MCP membrane
3. Surface composition

Phase 3:
1. Schema fetching over WebSocket
2. Full synapse port
3. Web membrane

### Testing Strategy

```haskell
-- Property: parse and render are inverses for valid inputs
prop_parseRenderRoundtrip :: MethodSchema -> Value -> Property
prop_parseRenderRoundtrip m val =
  let membrane = CLIMembrane m
      rendered = render membrane val
  in case parse membrane rendered of
       Right parsed -> parsed === val
       Left _ -> property False  -- Should not fail

-- Unit tests for each TypeSchema variant
spec_TString_parse :: Spec
spec_TString_parse = do
  it "parses plain strings" $ ...
  it "validates enum values" $ ...
  it "respects format constraints" $ ...
```

### Integration with Synapse

Once membrane is built, synapse becomes:

```haskell
-- synapse/Main.hs (entire file)
module Main where

import Membrane
import Transport (connectSubstrate, fetchSchema)

main :: IO ()
main = do
  conn <- connectSubstrate "ws://127.0.0.1:4444"
  schemas <- fetchSchema conn
  let surface = buildSurface schemas
  runCLI conn surface
```

All the complexity lives in membrane. Synapse is just transport + orchestration.

## References

- [Bidirectional Transformations](https://www.cis.upenn.edu/~bcpierce/papers/lenses-etapsslides.pdf) - Lenses in PL theory
- [Servant (Haskell)](https://docs.servant.dev/) - Type-level API definitions that generate clients, servers, and docs
- [optparse-applicative](https://hackage.haskell.org/package/optparse-applicative) - Declarative CLI parsing (used by synapse)
