# Loom Deletion Policy & Reference Counting

## Overview

Loom implements a sophisticated deletion policy based on reference counting and gradual archival. Nothing is truly deleted immediately—instead, resources transition through states that allow recovery and ownership transfer.

## Deletion States

Resources in Loom transition through several states:

```
ACTIVE → SCHEDULED_DELETE → ARCHIVED → PURGED
```

### 1. ACTIVE
- Default state for all resources
- Fully accessible and mutable
- Reference count ≥ 1

### 2. SCHEDULED_DELETE
- Marked for deletion but not yet archived
- Reference count = 0
- Still fully accessible for reads
- Any system can "claim" ownership (increment ref count → returns to ACTIVE)
- Duration: Configurable (default: 7 days)
- Operations that work:
  - `tree_get`, `node_get` (read operations)
  - `tree_claim`, `node_claim` (increment ref count)
  - Queries return with `deletion_scheduled: true` flag

### 3. ARCHIVED
- Beyond the scheduled deletion window
- Read-only access via special archive queries
- Cannot be claimed (returns error)
- Duration: Configurable (default: 30 days)
- Operations that work:
  - `tree_get_archived`, `node_get_archived`
  - These return with `archived: true` flag

### 4. PURGED
- Completely removed from database
- No recovery possible
- Happens automatically after archive period expires

## Reference Counting

Each tree and node has a reference count:

```rust
struct ResourceRefs {
    /// Total reference count
    ref_count: i64,

    /// Who owns references (for debugging/auditing)
    owners: HashMap<String, i64>, // owner_id -> count
}
```

### Operations

**Increment Reference (`claim`)**:
```rust
loom.tree_claim(tree_id: TreeId, owner_id: String, count: i64 = 1)
loom.node_claim(tree_id: TreeId, node_id: NodeId, owner_id: String, count: i64 = 1)
```

**Decrement Reference (`release`)**:
```rust
loom.tree_release(tree_id: TreeId, owner_id: String, count: i64 = 1)
loom.node_release(tree_id: TreeId, node_id: NodeId, owner_id: String, count: i64 = 1)
```

**Schedule Deletion**:
- When ref_count reaches 0, automatically transitions to SCHEDULED_DELETE
- Sets `scheduled_deletion_at` timestamp

**Force Deletion** (admin only):
```rust
loom.tree_force_delete(tree_id: TreeId) // Immediately moves to ARCHIVED
```

## Handle Versioning

Handles now include version information to support frontend rendering:

```rust
struct Handle {
    /// Source system identifier
    source: String,

    /// Source system version (semantic version)
    source_version: String,

    /// Identifier within that source system
    identifier: String,

    /// Optional metadata for the handle
    metadata: Option<Value>,
}
```

### Example Handles

**Bash Plugin (v1.0.0)**:
```json
{
  "source": "bash",
  "source_version": "1.0.0",
  "identifier": "exec_550e8400",
  "metadata": {
    "command": "echo hello",
    "exit_code": 0
  }
}
```

**Future: Postgres Storage Plugin (v2.1.3)**:
```json
{
  "source": "postgres",
  "source_version": "2.1.3",
  "identifier": "message_12345",
  "metadata": {
    "content_type": "text/markdown",
    "schema_version": "2"
  }
}
```

The frontend can use `source` and `source_version` to:
1. Select the appropriate renderer component
2. Handle version-specific data formats
3. Show warnings for deprecated versions

## Plugin Versioning

All plugins must implement versioning:

```rust
trait Plugin {
    fn namespace(&self) -> &str;
    fn version(&self) -> &str; // Semantic version: "MAJOR.MINOR.PATCH"
    fn methods(&self) -> Vec<&str>;
    // ... other methods
}
```

The hub wraps plugin responses with version information:

```json
{
  "event": "data",
  "data": {
    "path": { "segments": ["bash"] },
    "content_type": "bash.event",
    "plugin_version": "1.0.0",
    "data": { ... }
  }
}
```

## Database Schema

```sql
-- Trees table
CREATE TABLE trees (
    id TEXT PRIMARY KEY,
    root_node_id TEXT NOT NULL,
    ref_count INTEGER NOT NULL DEFAULT 1,
    state TEXT NOT NULL DEFAULT 'active', -- 'active', 'scheduled_delete', 'archived'
    scheduled_deletion_at INTEGER, -- Unix timestamp
    archived_at INTEGER, -- Unix timestamp
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL,
    metadata TEXT -- JSON
);

-- Tree reference owners
CREATE TABLE tree_refs (
    tree_id TEXT NOT NULL,
    owner_id TEXT NOT NULL,
    count INTEGER NOT NULL DEFAULT 1,
    claimed_at INTEGER NOT NULL,
    PRIMARY KEY (tree_id, owner_id),
    FOREIGN KEY (tree_id) REFERENCES trees(id)
);

-- Nodes table
CREATE TABLE nodes (
    id TEXT PRIMARY KEY,
    tree_id TEXT NOT NULL,
    parent_id TEXT, -- NULL for root
    ref_count INTEGER NOT NULL DEFAULT 1,
    state TEXT NOT NULL DEFAULT 'active',
    scheduled_deletion_at INTEGER,
    archived_at INTEGER,
    node_type TEXT NOT NULL, -- 'text' or 'external'
    -- For text nodes
    content TEXT,
    -- For external nodes
    handle_source TEXT,
    handle_source_version TEXT,
    handle_identifier TEXT,
    handle_metadata TEXT, -- JSON
    created_at INTEGER NOT NULL,
    metadata TEXT, -- JSON
    FOREIGN KEY (tree_id) REFERENCES trees(id)
);

-- Node reference owners
CREATE TABLE node_refs (
    node_id TEXT NOT NULL,
    owner_id TEXT NOT NULL,
    count INTEGER NOT NULL DEFAULT 1,
    claimed_at INTEGER NOT NULL,
    PRIMARY KEY (node_id, owner_id),
    FOREIGN KEY (node_id) REFERENCES nodes(id)
);

-- Node children (adjacency list)
CREATE TABLE node_children (
    parent_id TEXT NOT NULL,
    child_id TEXT NOT NULL,
    position INTEGER NOT NULL, -- Order of children
    PRIMARY KEY (parent_id, child_id),
    FOREIGN KEY (parent_id) REFERENCES nodes(id),
    FOREIGN KEY (child_id) REFERENCES nodes(id)
);

-- Indices for performance
CREATE INDEX idx_trees_state ON trees(state);
CREATE INDEX idx_trees_scheduled_deletion ON trees(scheduled_deletion_at) WHERE state = 'scheduled_delete';
CREATE INDEX idx_trees_archived ON trees(archived_at) WHERE state = 'archived';

CREATE INDEX idx_nodes_tree ON nodes(tree_id);
CREATE INDEX idx_nodes_parent ON nodes(parent_id);
CREATE INDEX idx_nodes_state ON nodes(state);
CREATE INDEX idx_nodes_scheduled_deletion ON nodes(scheduled_deletion_at) WHERE state = 'scheduled_delete';

CREATE INDEX idx_node_children_parent ON node_children(parent_id);
CREATE INDEX idx_node_children_child ON node_children(child_id);
```

## Configuration

```rust
struct LoomConfig {
    /// Duration before scheduled resources move to archived (seconds)
    scheduled_deletion_window: i64, // Default: 7 days = 604800

    /// Duration before archived resources are purged (seconds)
    archive_window: i64, // Default: 30 days = 2592000

    /// Path to SQLite database
    db_path: PathBuf,

    /// Enable auto-cleanup background task
    auto_cleanup: bool, // Default: true

    /// Cleanup task interval (seconds)
    cleanup_interval: i64, // Default: 1 hour = 3600
}
```

## Auto-Cleanup Task

A background task runs periodically to:

1. **Transition SCHEDULED_DELETE → ARCHIVED**:
   ```sql
   UPDATE trees
   SET state = 'archived', archived_at = ?
   WHERE state = 'scheduled_delete'
   AND scheduled_deletion_at <= ?
   ```

2. **Purge ARCHIVED resources**:
   ```sql
   DELETE FROM trees
   WHERE state = 'archived'
   AND archived_at <= ?
   ```

## API Operations

### Reference Management

```rust
// Claim ownership (increment ref count)
loom.tree_claim(tree_id: TreeId, owner_id: String, count: i64 = 1) -> Result<()>
loom.node_claim(tree_id: TreeId, node_id: NodeId, owner_id: String, count: i64 = 1) -> Result<()>

// Release ownership (decrement ref count)
loom.tree_release(tree_id: TreeId, owner_id: String, count: i64 = 1) -> Result<()>
loom.node_release(tree_id: TreeId, node_id: NodeId, owner_id: String, count: i64 = 1) -> Result<()>

// Get reference info
loom.tree_get_refs(tree_id: TreeId) -> Result<ResourceRefs>
loom.node_get_refs(tree_id: TreeId, node_id: NodeId) -> Result<ResourceRefs>
```

### Deletion Query Operations

```rust
// List resources scheduled for deletion (can be claimed)
loom.tree_list_scheduled() -> Vec<TreeId>
loom.node_list_scheduled(tree_id: TreeId) -> Vec<NodeId>

// Get archived resources (read-only)
loom.tree_get_archived(tree_id: TreeId) -> Result<Tree>
loom.node_get_archived(tree_id: TreeId, node_id: NodeId) -> Result<Node>

// List archived resources
loom.tree_list_archived() -> Vec<TreeId>
```

### Admin Operations

```rust
// Force immediate archival (admin only)
loom.tree_force_archive(tree_id: TreeId) -> Result<()>

// Force immediate purge (admin only, dangerous!)
loom.tree_force_purge(tree_id: TreeId) -> Result<()>
```

## Event Stream Updates

Events now include reference counting information:

```rust
enum LoomEvent {
    // Reference events
    TreeClaimed { tree_id: TreeId, owner_id: String, new_count: i64 },
    TreeReleased { tree_id: TreeId, owner_id: String, new_count: i64 },
    TreeScheduledDeletion { tree_id: TreeId, scheduled_at: i64 },
    TreeArchived { tree_id: TreeId, archived_at: i64 },
    TreePurged { tree_id: TreeId },

    // Node reference events
    NodeClaimed { tree_id: TreeId, node_id: NodeId, owner_id: String, new_count: i64 },
    NodeReleased { tree_id: TreeId, node_id: NodeId, owner_id: String, new_count: i64 },
    NodeScheduledDeletion { tree_id: TreeId, node_id: NodeId, scheduled_at: i64 },
    NodeArchived { tree_id: TreeId, node_id: NodeId, archived_at: i64 },
    NodePurged { tree_id: TreeId, node_id: NodeId },

    // Existing events...
    TreeCreated { tree_id: TreeId },
    TreeData { tree: Tree, state: ResourceState, refs: ResourceRefs },
    // ...
}

struct ResourceState {
    state: String, // "active", "scheduled_delete", "archived"
    scheduled_deletion_at: Option<i64>,
    archived_at: Option<i64>,
}
```

## Usage Examples

### Creating and Using a Tree

```rust
// Create tree (automatically has ref_count = 1, owner = "system")
let tree_id = loom.tree_create(None);

// Another system claims it
loom.tree_claim(tree_id, "conversation_manager", 1);
// ref_count = 2, owners = {"system": 1, "conversation_manager": 1}

// System releases
loom.tree_release(tree_id, "system", 1);
// ref_count = 1, owners = {"conversation_manager": 1}

// Conversation manager releases
loom.tree_release(tree_id, "conversation_manager", 1);
// ref_count = 0, state = "scheduled_delete"
// scheduled_deletion_at = now + 7 days
```

### Rescuing a Scheduled Resource

```rust
// List trees scheduled for deletion
let scheduled = loom.tree_list_scheduled();

// User wants to keep one
loom.tree_claim(scheduled[0], "user_123", 1);
// State returns to "active", scheduled_deletion_at = NULL
```

### Querying Archived Data

```rust
// Normal query fails
let result = loom.tree_get(old_tree_id);
// Error: "Tree is archived, use tree_get_archived()"

// Archive query works
let tree = loom.tree_get_archived(old_tree_id)?;
// Returns tree with archived: true flag

// Trying to claim fails
loom.tree_claim(old_tree_id, "user", 1);
// Error: "Cannot claim archived resource"
```

## Benefits

1. **No Accidental Data Loss**: 7-day grace period before archival
2. **Ownership Transfer**: Other systems can rescue scheduled deletions
3. **Audit Trail**: Track who owns references via `tree_refs` table
4. **Gradual Cleanup**: Resources transition through states gracefully
5. **Frontend-Friendly**: Version information enables proper rendering
6. **Persistence**: SQLite backend ensures durability
7. **Performance**: Indices optimize common queries

## Migration Strategy

When plugin versions change:

1. **Minor Version**: Backwards compatible, old handles still work
2. **Major Version**: May break compatibility
   - Frontend checks `source_version` in handle
   - Shows migration UI or compatibility layer
   - Option to "upgrade" handle to new version

This is handled at the application layer, not in Loom itself.
