# Loom: Context Management Architecture

## Overview

**Loom** is the context management plugin that provides tree-based conversation history. It's source-agnostic, meaning it doesn't store actual data—only handles pointing to where data lives in external systems.

## Key Concepts

### Terminology

- **Loom**: The context management system (the plugin)
- **Conversation Tree**: The tree structure representing branching conversation paths
- **Context**: A specific path through a conversation tree (root to leaf)
- **Node**: A single point in the conversation tree
- **Handle**: A reference to data stored elsewhere (source + identifier)
- **Tree**: A collection of connected nodes with a single root

### Design Principles

1. **Source Agnostic**: Loom doesn't know or care where data actually lives
2. **Handle-Based**: Nodes store handles, not content
3. **Immutable**: Operations create new nodes rather than mutating existing ones
4. **Lightweight Navigation**: Can download tree structure without fetching actual data
5. **CRUD Complete**: Full Create, Read, Update, Delete operations

## Data Model

### Core Types

```rust
/// Unique identifier for a tree
type TreeId = Uuid;

/// Unique identifier for a node within a tree
type NodeId = Uuid;

/// Handle pointing to external data with versioning
struct Handle {
    /// Source system identifier (e.g., "postgres", "s3", "redis", "bash")
    source: String,

    /// Source system version (semantic version: "MAJOR.MINOR.PATCH")
    source_version: String,

    /// Identifier within that source system
    identifier: String,

    /// Optional metadata for the handle (e.g., content type)
    metadata: Option<serde_json::Value>,
}

/// Node type discriminator
enum NodeType {
    /// Built-in text node (data stored in Loom)
    Text { content: String },

    /// External data reference
    External { handle: Handle },
}

/// A node in the conversation tree
struct Node {
    /// Unique identifier for this node
    id: NodeId,

    /// Parent node (None for root)
    parent: Option<NodeId>,

    /// Child nodes
    children: Vec<NodeId>,

    /// Node data (handle or built-in)
    data: NodeType,

    /// Creation timestamp
    created_at: i64,

    /// Optional metadata
    metadata: Option<serde_json::Value>,
}

/// A conversation tree
struct Tree {
    /// Unique identifier for this tree
    id: TreeId,

    /// Root node ID
    root: NodeId,

    /// All nodes in the tree (NodeId -> Node)
    nodes: HashMap<NodeId, Node>,

    /// Creation timestamp
    created_at: i64,

    /// Last modified timestamp
    updated_at: i64,

    /// Optional tree-level metadata (name, description, etc.)
    metadata: Option<serde_json::Value>,
}
```

### Lightweight Tree Structure

For efficient navigation without fetching full data:

```rust
/// Lightweight node representation (just structure, no data)
struct NodeSkeleton {
    id: NodeId,
    parent: Option<NodeId>,
    children: Vec<NodeId>,
    /// Type indicator (but not the actual data)
    node_type: String, // "text" or "external"
}

/// Lightweight tree structure
struct TreeSkeleton {
    id: TreeId,
    root: NodeId,
    nodes: HashMap<NodeId, NodeSkeleton>,
}
```

## API Operations

### Tree Operations

```rust
// CREATE
loom.tree_create(metadata: Option<Value>) -> TreeId

// READ
loom.tree_get(tree_id: TreeId) -> Tree
loom.tree_get_skeleton(tree_id: TreeId) -> TreeSkeleton
loom.tree_list() -> Vec<TreeId>

// UPDATE
loom.tree_update_metadata(tree_id: TreeId, metadata: Value) -> ()

// DELETE
loom.tree_delete(tree_id: TreeId) -> ()
```

### Node Operations

```rust
// CREATE
loom.node_create_text(
    tree_id: TreeId,
    parent: Option<NodeId>,
    content: String,
    metadata: Option<Value>
) -> NodeId

loom.node_create_external(
    tree_id: TreeId,
    parent: Option<NodeId>,
    handle: Handle,
    metadata: Option<Value>
) -> NodeId

// READ
loom.node_get(tree_id: TreeId, node_id: NodeId) -> Node
loom.node_get_children(tree_id: TreeId, node_id: NodeId) -> Vec<NodeId>
loom.node_get_parent(tree_id: TreeId, node_id: NodeId) -> Option<NodeId>
loom.node_get_path(tree_id: TreeId, node_id: NodeId) -> Vec<NodeId> // root to node

// UPDATE (creates new node, doesn't mutate)
loom.node_update(
    tree_id: TreeId,
    node_id: NodeId,
    data: NodeType,
    metadata: Option<Value>
) -> NodeId // returns NEW node ID

// DELETE (removes from tree structure)
loom.node_delete(tree_id: TreeId, node_id: NodeId) -> ()
```

### Context Operations

A context is a specific path from root to a leaf node:

```rust
// Get all leaf nodes in a tree
loom.context_list_leaves(tree_id: TreeId) -> Vec<NodeId>

// Get the full path (context) from root to a specific node
loom.context_get_path(tree_id: TreeId, node_id: NodeId) -> Vec<Node>

// Get just the handles along a path (lightweight)
loom.context_get_handles(tree_id: TreeId, node_id: NodeId) -> Vec<Handle>
```

## Plugin Events

The Loom plugin emits typed events:

```rust
enum LoomEvent {
    // Tree events
    TreeCreated { tree_id: TreeId },
    TreeDeleted { tree_id: TreeId },
    TreeUpdated { tree_id: TreeId },

    // Node events
    NodeCreated { tree_id: TreeId, node_id: NodeId, parent: Option<NodeId> },
    NodeUpdated { tree_id: TreeId, old_id: NodeId, new_id: NodeId },
    NodeDeleted { tree_id: TreeId, node_id: NodeId },

    // Data events
    NodeData { tree_id: TreeId, node_id: NodeId, data: NodeType },
    TreeStructure { skeleton: TreeSkeleton },
    ContextPath { tree_id: TreeId, path: Vec<NodeId> },
}
```

## Storage

Loom uses SQLite for persistent storage with reference counting and soft deletion.

See [Loom Deletion Policy](./16681472986707949567_loom-deletion-policy.md) for complete details on:
- Reference counting and ownership tracking
- Gradual deletion states (ACTIVE → SCHEDULED_DELETE → ARCHIVED → PURGED)
- Handle versioning for frontend rendering
- Database schema and indices

```rust
struct LoomStorage {
    /// SQLite connection pool
    pool: SqlitePool,

    /// Configuration
    config: LoomConfig,
}

struct LoomConfig {
    /// Duration before scheduled resources move to archived (seconds)
    scheduled_deletion_window: i64, // Default: 7 days

    /// Duration before archived resources are purged (seconds)
    archive_window: i64, // Default: 30 days

    /// Path to SQLite database
    db_path: PathBuf,

    /// Enable auto-cleanup background task
    auto_cleanup: bool,
}
```

## Usage Examples

### Creating a Conversation Tree

```rust
// Create a tree
let tree_id = loom.tree_create(Some(json!({ "name": "My Conversation" })));

// Create root node (text)
let root_id = loom.node_create_text(
    tree_id,
    None,
    "Hello, how can I help you?",
    None
);

// Create child node (external handle to user message)
let user_msg_id = loom.node_create_external(
    tree_id,
    Some(root_id),
    Handle {
        source: "postgres".into(),
        identifier: "message_12345".into(),
        metadata: Some(json!({ "content_type": "text/plain" })),
    },
    None
);

// Branch: create two different responses
let response_a = loom.node_create_text(
    tree_id,
    Some(user_msg_id),
    "Response A: Let me explain...",
    None
);

let response_b = loom.node_create_text(
    tree_id,
    Some(user_msg_id),
    "Response B: Here's another way...",
    None
);
```

### Navigating the Tree

```rust
// Get lightweight structure first
let skeleton = loom.tree_get_skeleton(tree_id);

// Find all leaf nodes (conversation endpoints)
let leaves = loom.context_list_leaves(tree_id);

// Get the full context path for a specific leaf
let path = loom.context_get_path(tree_id, response_a);
// Returns: [root, user_msg, response_a]

// Get just handles (for fetching from source systems)
let handles = loom.context_get_handles(tree_id, response_a);
// Returns handles for any external nodes in the path
```

### Updating a Node (Immutable)

```rust
// "Update" by creating a new node
let updated_node_id = loom.node_update(
    tree_id,
    response_a,
    NodeType::Text { content: "Updated response A".into() },
    Some(json!({ "edited": true }))
);

// Old node (response_a) still exists but is orphaned
// New node (updated_node_id) has taken its place in the tree
```

## Implementation Phases

### Phase 1: Core Types & In-Memory Storage
- Define Rust types
- Implement in-memory storage
- Basic CRUD operations

### Phase 2: Plugin Integration
- Implement as hub plugin
- Stream events for operations
- JSON-RPC method handlers

### Phase 3: Haskell Client
- Define typed Haskell API
- Event stream handling
- CLI commands for tree navigation

### Phase 4: Persistence
- File-based persistence (JSON)
- Optional SQLite backend

### Phase 5: Super Loom (Future)
- Cross-tree operations
- Advanced queries
- Tree merging/diffing

## Wire Format

### Request Format

```json
{
  "jsonrpc": "2.0",
  "method": "loom_tree_create",
  "params": {
    "metadata": { "name": "My Conversation" }
  },
  "id": 1
}
```

### Response Format (Subscription)

```json
{
  "jsonrpc": "2.0",
  "method": "subscription",
  "params": {
    "subscription": "sub_123",
    "result": {
      "event": "data",
      "data": {
        "path": { "segments": ["loom"] },
        "content_type": "loom.event",
        "data": {
          "type": "tree_created",
          "data": {
            "tree_id": "550e8400-e29b-41d4-a716-446655440000"
          }
        }
      }
    }
  }
}
```

## Design Rationale

### Why Handles?

1. **Separation of Concerns**: Loom manages structure, not content
2. **Flexibility**: Content can live anywhere (DB, S3, cache, etc.)
3. **Performance**: Tree structure can be fetched without expensive data loads
4. **Scalability**: Large conversations don't bloat tree storage

### Why Immutable Updates?

1. **History Preservation**: Old versions remain accessible
2. **Time Travel**: Can reconstruct any point in conversation history
3. **Branching**: Natural support for exploring alternatives
4. **Simplicity**: No complex mutation logic

### Why Built-in Text Nodes?

1. **Convenience**: Simple messages don't need external storage
2. **Bootstrapping**: Can use Loom without any other systems
3. **Performance**: No round-trip for simple data

## Security Considerations

- **Handle Validation**: Source systems must validate handle access
- **Tree Ownership**: Future: associate trees with users/sessions
- **Rate Limiting**: Prevent tree creation spam
- **Size Limits**: Max nodes per tree, max tree depth

## Future Extensions

- **Tree Templates**: Predefined structures for common patterns
- **Tree Queries**: Find nodes by metadata, content type, etc.
- **Tree Statistics**: Depth, breadth, leaf count, etc.
- **Node Attachments**: Additional metadata, tags, annotations
- **Tree Merging**: Combine trees or branches
- **Export/Import**: Serialize/deserialize entire trees
