# Testing Strategy

## Overview

Substrate uses a multi-layered testing approach with three distinct test categories:
1. **Unit tests** - Test individual components in isolation
2. **Integration tests** - Test JSON-RPC endpoints with a running server
3. **Schema validation tests** - Verify type-driven schema generation

This document describes what each test layer validates, patterns used, and how they work together to ensure correctness.

## Test Layers

### 1. Unit Tests (In-Module)

Unit tests live alongside the code they test using Rust's `#[cfg(test)]` module pattern.

#### Method Enum Tests (`src/activations/*/methods.rs`)

Test that method enums are correctly structured for schema generation.

**Location**: Each activation's `methods.rs` file
**Run with**: `cargo test` (runs automatically)

**What they test**:
```rust
#[test]
fn test_method_names() {
    // Verify method.name() returns correct string
    let method = ArborMethod::TreeCreate { ... };
    assert_eq!(method.name(), "tree_create");
}

#[test]
fn test_serialize() {
    // Verify serde serialization uses correct method names
    let method = ArborMethod::TreeList;
    let json = serde_json::to_string(&method).unwrap();
    assert!(json.contains("tree_list"));
}

#[test]
fn test_schema_has_uuid_format() {
    // Verify uuid::Uuid fields get format: "uuid" annotation
    let schema = ArborMethod::schema();
    let schema_str = serde_json::to_string_pretty(&schema).unwrap();
    assert!(schema_str.contains("\"format\": \"uuid\""));
}

#[test]
fn test_schema_has_required_fields() {
    // Verify non-Option fields appear in required array
    let schema = ArborMethod::schema();
    let schema_str = serde_json::to_string(&schema).unwrap();
    assert!(schema_str.contains("\"required\""));
}

#[test]
fn test_schema_has_descriptions() {
    // Verify doc comments become schema descriptions
    let schema = ArborMethod::schema();
    let schema_str = serde_json::to_string(&schema).unwrap();
    assert!(schema_str.contains("\"description\""));
}
```

**Purpose**:
- Ensure schemars derives work correctly
- Catch schema regressions (missing required, format annotations)
- Verify type-driven schema generation produces expected output
- Document expected schema structure through tests

#### Executor Tests (`src/activations/bash/executor/mod.rs`)

Test activation core logic independent of RPC layer.

**What they test**:
```rust
#[tokio::test]
async fn test_execute_simple_command() {
    // Test bash execution produces stdout
    let executor = BashExecutor::new();
    let outputs = executor.execute_collect("echo 'hello world'").await;

    assert!(outputs.len() >= 2);  // stdout + exit
    match &outputs[0] {
        BashOutput::Stdout { line } => assert_eq!(line, "hello world"),
        _ => panic!("Expected stdout"),
    }
}

#[tokio::test]
async fn test_execute_stderr() {
    // Verify stderr is captured separately
    let outputs = executor.execute_collect("echo 'error' >&2").await;
    let has_stderr = outputs.iter().any(|o| matches!(o, BashOutput::Stderr { .. }));
    assert!(has_stderr);
}

#[tokio::test]
async fn test_execute_exit_code() {
    // Verify exit codes are captured
    let outputs = executor.execute_collect("exit 42").await;
    match outputs.last().unwrap() {
        BashOutput::Exit { code } => assert_eq!(*code, 42),
        _ => panic!("Expected exit"),
    }
}
```

**Purpose**:
- Test business logic without RPC/networking overhead
- Fast feedback loop during development
- Verify edge cases (stderr, exit codes, errors)

#### Guided Error Tests (`src/plexus/errors.rs`)

Test error enrichment system.

**What they test**:
```rust
#[test]
fn test_parse_error_includes_try() {
    // Verify parse errors include "try" guidance
    let error = GuidedError::parse_error("invalid JSON");
    let data: GuidedErrorData = serde_json::from_str(...).unwrap();
    assert_eq!(data.try_request.method, "plexus_schema");
}

#[test]
fn test_activation_not_found_includes_available() {
    // Verify unknown activation errors list available ones
    let error = GuidedError::activation_not_found(
        "foo",
        vec!["arbor".into(), "bash".into(), "health".into()],
    );
    let data: GuidedErrorData = serde_json::from_str(...).unwrap();
    assert_eq!(data.try_request.method, "plexus_schema");
    assert_eq!(data.context["available_activations"].as_array().unwrap().len(), 3);
}

#[test]
fn test_method_not_found_with_example() {
    // Verify method errors include executable examples
    let error = GuidedError::method_not_found(
        "bash", "foo", vec!["execute".into()],
        Some(("bash_execute", vec![json!("echo hello")])),
    );
    let data: GuidedErrorData = serde_json::from_str(...).unwrap();
    assert_eq!(data.try_request.method, "bash_execute");
    assert_eq!(data.try_request.params[0], json!("echo hello"));
}
```

**Purpose**:
- Ensure error responses help users fix mistakes
- Verify guided error structure is correct
- Test error enrichment middleware behavior

#### Schema Utility Tests (`src/plexus/schema.rs`)

Test schema manipulation utilities.

**What they test**:
- Schema serialization/deserialization
- Helper methods like `get_method_schema()`, `list_methods()`
- Schema structure validation

**Purpose**:
- Verify schema utilities work correctly
- Test method extraction from oneOf schemas
- Ensure schema type conversions preserve data

### 2. Integration Tests (`tests/rpc_integration.rs`)

Integration tests verify end-to-end JSON-RPC functionality with a live server.

**Requirements**:
- Server must be running: `cargo run`
- Default port: 4444 (override with `SUBSTRATE_PORT`)
- Run with: `cargo test --test rpc_integration`

**Test categories**:

#### Schema Query Tests

Verify schema endpoints return correct structure.

```rust
#[tokio::test]
async fn test_schema_includes_required_in_params() {
    // Query arbor activation schema
    let mut subscription = client.subscribe::<Value, _>(
        "plexus_activation_schema",
        rpc_params!["arbor"],
        "unsubscribe_schema"
    ).await.expect(...);

    // Navigate to node_create_text variant
    let params = find_method_params("node_create_text");
    let required = params.get("required").expect("params should have 'required' field");

    // Verify required fields
    assert!(required.contains(&"tree_id"));
    assert!(required.contains(&"content"));
    assert!(!required.contains(&"parent"));  // Optional field
}
```

**Tests**:
- `test_schema_includes_required_in_params` - Full activation schema
- `test_method_level_schema_query` - Single method schema query
- `test_method_schema_unknown_method` - Error handling for unknown methods

**Purpose**:
- Verify wrapper pattern generates schemas correctly
- Test progressive disclosure (activation → method → params)
- Ensure schema structure matches expectations

#### Plexus Hash Tests

Verify cache invalidation hash is included in all responses.

```rust
#[tokio::test]
async fn test_responses_include_plexus_hash() {
    // Test health_check includes hash
    let event = subscribe_and_get_event("health_check", rpc_params![]).await;
    let plexus_hash = event.get("plexus_hash").expect("Should have plexus_hash");
    assert!(plexus_hash.is_string() && !plexus_hash.as_str().unwrap().is_empty());

    // Test plexus_hash endpoint itself
    let event = subscribe_and_get_event("plexus_hash", rpc_params![]).await;
    let hash_top = event.get("plexus_hash").unwrap();
    let hash_data = event.get("data").unwrap().get("hash").unwrap();
    assert_eq!(hash_top, hash_data);  // Consistency check
}
```

**Purpose**:
- Ensure all responses include cache invalidation hash
- Verify hash consistency across endpoints
- Test hash generation and middleware injection

#### Guided Error Tests

Verify error responses include helpful guidance.

```rust
#[tokio::test]
async fn test_guided_error_includes_try_field() {
    // Try unknown activation 'foo'
    let result = client.subscribe::<Value, _>(
        "foo_bar",
        rpc_params![],
        "unsubscribe_foo"
    ).await;

    let err = result.expect_err("Should fail for unknown activation");
    let err_str = format!("{:?}", err);

    // Verify error includes guidance
    assert!(err_str.contains("Activation 'foo' not found"));
    assert!(err_str.contains("plexus_schema"));  // Suggests how to discover
    assert!(err_str.contains("available_activations"));
    assert!(err_str.contains("arbor") && err_str.contains("bash"));
}
```

**Tests**:
- `test_guided_error_unknown_activation` - Basic error detection
- `test_guided_error_includes_try_field` - Error enrichment with suggestions

**Purpose**:
- Verify guided error middleware enriches errors
- Test error responses help users discover correct usage
- Ensure error structure includes actionable guidance

#### Activation Functional Tests

Test each activation's core workflows.

**Health activation**:
```rust
#[tokio::test]
async fn test_health_check() {
    let event = subscribe_and_get_event("health_check", rpc_params![]).await;
    let data = event.get("data").expect("Should have data");
    assert_eq!(data.get("status").and_then(|v| v.as_str()), Some("healthy"));
}
```

**Bash activation**:
```rust
#[tokio::test]
async fn test_bash_execute() {
    let mut subscription = client.subscribe(
        "bash_execute",
        rpc_params!["echo hello"],
        "unsubscribe_execute"
    ).await.expect(...);

    // Collect all events
    let mut events = Vec::new();
    while let Ok(Some(result)) = tokio::time::timeout(..., subscription.next()).await {
        events.push(result.expect("Error receiving event"));
    }

    // Verify stdout contains "hello"
    let found_output = events.iter().any(|e| {
        e.get("data")
            .and_then(|d| d.get("line"))
            .and_then(|v| v.as_str())
            .map(|s| s.contains("hello"))
            .unwrap_or(false)
    });
    assert!(found_output);
}
```

**Arbor activation**:
```rust
#[tokio::test]
async fn test_arbor_full_workflow() {
    // 1. Create tree
    let tree_id = create_tree(metadata, owner_id).await;

    // 2. Get tree to find root
    let root_id = get_tree(tree_id).await.root;

    // 3. Create text node
    let node_id = create_text_node(tree_id, root_id, "Hello, world!", metadata).await;

    // 4. Get node and verify content
    let node = get_node(tree_id, node_id).await;
    assert_eq!(node.content, "Hello, world!");

    // 5. Get context path
    let path = get_context_path(tree_id, node_id).await;
    assert!(path.nodes.len() >= 2);  // Root + our node
}
```

**Cone activation**:
```rust
#[tokio::test]
async fn test_cone_create() {
    let event = subscribe_and_get_event(
        "cone_create",
        rpc_params!["test-cone", "gpt-4o-mini", "You are a test assistant.", json!(null)]
    ).await;

    let data = event.get("data").expect("Should have data");
    assert!(data.get("cone_id").is_some() || data.get("agent_id").is_some());
}

#[tokio::test]
async fn test_cone_registry() {
    let event = subscribe_and_get_event("cone_registry", rpc_params![]).await;
    let data = event.get("data").expect("Should have data");

    // Verify structure from cllient::ModelExport
    let services = data.get("services").expect("Should have services array");
    let families = data.get("families").expect("Should have families array");
    let models = data.get("models").expect("Should have models array");
    let stats = data.get("stats").expect("Should have stats");

    // Verify model structure
    let first_model = &models.as_array().unwrap()[0];
    assert!(first_model.get("id").is_some());
    assert!(first_model.get("family").is_some());
    assert!(first_model.get("capabilities").is_some());
    assert!(first_model.get("pricing").is_some());
}
```

**Purpose**:
- Verify activations work end-to-end via RPC
- Test realistic workflows (create → get → update)
- Ensure data flows correctly through all layers
- Catch integration issues between components

## Test Execution

### Running All Tests

```bash
# Run unit tests
cargo test

# Run integration tests (requires running server)
cargo run  # In one terminal
cargo test --test rpc_integration  # In another terminal
```

### Running Specific Test Categories

```bash
# Only method schema tests
cargo test test_schema

# Only executor tests
cargo test executor::tests

# Only guided error tests
cargo test plexus::errors::tests

# Specific integration test
cargo test test_arbor_full_workflow
```

## Test Patterns and Conventions

### Unit Test Patterns

1. **Inline modules**: Tests live in `#[cfg(test)] mod tests { }`
2. **Descriptive names**: `test_schema_has_uuid_format` not `test_schema1`
3. **Clear assertions**: Include context in assertion messages
4. **Minimal setup**: Tests should be easy to understand in isolation

### Integration Test Patterns

1. **Timeout protection**: All subscription reads use `tokio::time::timeout`
2. **Helper functions**: `create_client()`, `subscribe_and_get_event()` reduce duplication
3. **Clear failure messages**: Assertions include debug output for failures
4. **Event structure validation**: Navigate JSON carefully with error messages
5. **Emoji indicators**: `eprintln!("✓ Test passed with details")` for visibility

### Common Test Utilities

```rust
// Create WebSocket client
async fn create_client() -> Result<WsClient, Box<dyn Error>> {
    WsClientBuilder::default()
        .connection_timeout(Duration::from_secs(5))
        .build(&server_url())
        .await
}

// Get server URL from environment
fn server_url() -> String {
    let port = std::env::var("SUBSTRATE_PORT").unwrap_or_else(|_| "4444".to_string());
    format!("ws://127.0.0.1:{}", port)
}
```

## What Tests Don't Cover (Yet)

Areas that could benefit from additional testing:

1. **Concurrency**: Multiple concurrent subscriptions to same method
2. **Error paths**: More edge cases in activation error handling
3. **Stream completeness**: Verify all expected events arrive in order
4. **Performance**: Load testing, stress testing
5. **Tree operations**: Complex arbor tree manipulations
6. **Cone conversations**: Multi-turn conversation flows
7. **Reference counting**: Arbor ref count invariants
8. **Schema evolution**: Versioning and backwards compatibility

## Benefits of This Approach

### Fast Feedback Loop

- Unit tests run in milliseconds
- No server startup overhead
- Isolated failures are easy to debug

### Comprehensive Coverage

- Unit tests verify components work correctly
- Integration tests verify components work together
- Schema tests ensure type safety and documentation

### Regression Protection

- Schema changes trigger test failures
- API changes are caught immediately
- Refactoring is safe with good test coverage

### Documentation

Tests serve as executable documentation:
- Show how to use each activation
- Demonstrate expected data structures
- Illustrate error handling patterns

### Development Workflow

```
1. Write unit test for new feature
2. Implement feature until unit test passes
3. Write integration test for end-to-end flow
4. Start server and verify integration test passes
5. Commit with passing tests
```

## Test File Locations

### Unit Tests
- `src/activations/*/methods.rs` - Method enum schema tests
- `src/activations/bash/executor/mod.rs` - Bash execution tests
- `src/plexus/errors.rs` - Guided error tests
- `src/plexus/schema.rs` - Schema utility tests

### Integration Tests
- `tests/rpc_integration.rs` - All JSON-RPC endpoint tests

### Test Data
Currently no fixtures needed - tests use:
- Inline test data (strings, JSON)
- SQLite in-memory databases (arbor, cone)
- Temporary test namespaces

## Related Documentation

- [Type-Driven Schema Generation](./16680887334149146623_type-driven-schema-generation.md) - How schemas are generated from types
- [Method-Level Schema Query](./16680890611072058623_method-level-schema-query.md) - Progressive schema discovery
- [Dynamic CLI Type-Driven Schemas](./16680891033387373567_dynamic-cli-type-driven-schemas.md) - CLI use of schemas
