Chaos Engineering

Renacer includes chaos engineering capabilities for testing system resilience under adverse conditions. This feature was added in Sprint 29, following patterns from the Aprender ML library.

Overview

Chaos engineering intentionally introduces controlled failures to verify system behavior under stress. For a syscall tracer like Renacer, this means testing how the tracer behaves when:

• Memory is limited
• CPU resources are constrained
• Processes exit unexpectedly
• Signals interrupt execution
• Network delays occur (for future network syscall analysis)

ChaosConfig Builder (Sprint 29)

Basic Usage

use renacer::chaos::ChaosConfig;
use std::time::Duration;

// Gentle chaos for regular testing
let config = ChaosConfig::gentle();

// Aggressive chaos for stress testing
let config = ChaosConfig::aggressive();

// Custom configuration
let config = ChaosConfig::new()
    .with_memory_limit(100 * 1024 * 1024)  // 100MB
    .with_cpu_limit(0.5)  // 50% CPU
    .with_timeout(Duration::from_secs(30))
    .with_signal_injection(true)
    .build();

Configuration Options

Tiered Chaos Features

Renacer's chaos capabilities are organized into progressive tiers, enabled via Cargo features:

Tier 1: Basic Chaos (chaos-basic)

Fast chaos - Resource limits and signal injection:

[dependencies]
renacer = { version = "0.4", features = ["chaos-basic"] }

Capabilities:

• Memory limit enforcement via cgroups
• CPU throttling
• Execution timeouts
• Signal injection (SIGINT, SIGTERM, SIGUSR1)

Use Cases:

• Testing error handling
• Validating graceful shutdowns
• Resource exhaustion scenarios

Tier 2: Network Chaos (chaos-network)

Network/IO chaos - Latency and packet loss simulation:

[dependencies]
renacer = { version = "0.4", features = ["chaos-network"] }

Capabilities:

• Simulated network latency (ms-level delays)
• Packet loss simulation
• Bandwidth throttling
• Connection drop simulation

Use Cases:

• Testing network syscall tracing
• Simulating slow I/O
• Network reliability testing

Tier 3: Byzantine Chaos (chaos-byzantine)

Byzantine fault injection - Syscall return modification:

[dependencies]
renacer = { version = "0.4", features = ["chaos-byzantine"] }

Capabilities:

• Modify syscall return values randomly
• Inject spurious errors (EINTR, EAGAIN)
• Simulate kernel bugs
• Corrupt data buffers

Use Cases:

• Testing error path robustness
• Validating retry logic
• Kernel bug simulation

Full Suite (chaos-full)

Complete chaos engineering - All features plus loom and arbitrary:

[dependencies]
renacer = { version = "0.4", features = ["chaos-full"] }

Additional Capabilities:

• Concurrency testing with loom
• Fuzzing integration with arbitrary
• Composite chaos scenarios

Builder Pattern (Aprender-Style)

Chainable API

let config = ChaosConfig::new()
    .with_memory_limit(64 * 1024 * 1024)
    .with_cpu_limit(0.3)
    .with_timeout(Duration::from_secs(15))
    .with_signal_injection(true)
    .with_network_latency(Duration::from_millis(50))
    .with_packet_loss_rate(0.05)  // 5% packet loss
    .build();

Preset Methods

// Gentle: Suitable for CI/CD
let gentle = ChaosConfig::gentle();
assert_eq!(gentle.memory_limit(), Some(500 * 1024 * 1024));
assert_eq!(gentle.cpu_limit(), Some(0.8));

// Aggressive: For stress testing
let aggressive = ChaosConfig::aggressive();
assert_eq!(aggressive.memory_limit(), Some(50 * 1024 * 1024));
assert_eq!(aggressive.signal_injection(), true);

Testing with Chaos

Property-Based Chaos Tests

Renacer includes 7 property-based tests for chaos configuration:

#[cfg(test)]
mod tests {
    use super::*;
    use proptest::prelude::*;

    proptest! {
        #[test]
        fn test_memory_limit_valid_range(limit in 1u64..10_000_000_000) {
            let config = ChaosConfig::new()
                .with_memory_limit(limit)
                .build();

            assert_eq!(config.memory_limit(), Some(limit));
        }

        #[test]
        fn test_cpu_limit_clamped(limit in any::<f64>()) {
            let config = ChaosConfig::new()
                .with_cpu_limit(limit)
                .build();

            // CPU limit should be clamped to [0.0, 1.0]
            if let Some(cpu) = config.cpu_limit() {
                assert!(cpu >= 0.0 && cpu <= 1.0);
            }
        }
    }
}

Integration Testing with Chaos

#[test]
fn test_tracer_under_memory_pressure() {
    let config = ChaosConfig::new()
        .with_memory_limit(10 * 1024 * 1024)  // 10MB limit
        .build();

    // Test that tracer handles OOM gracefully
    let result = run_tracer_with_chaos("ls", config);

    // Should either succeed or fail gracefully
    assert!(result.is_ok() || result.is_err());
    if let Err(e) = result {
        assert!(e.to_string().contains("memory"));
    }
}

Future CLI Integration

Planned for Sprint 30:

# Use gentle preset
renacer --chaos gentle -- ./app

# Use aggressive preset
renacer --chaos aggressive -- ./flaky-test

# Custom chaos from JSON
renacer --chaos custom:chaos.json -- ./stress-test

# Example chaos.json
{
  "memory_limit": 104857600,
  "cpu_limit": 0.5,
  "timeout_secs": 30,
  "signal_injection": true,
  "network_latency_ms": 100,
  "packet_loss_rate": 0.1
}

Use Cases

1. Testing Error Handling

// Verify tracer handles memory exhaustion
let config = ChaosConfig::new()
    .with_memory_limit(5 * 1024 * 1024)  // Very low limit
    .build();

// Tracer should fail gracefully, not crash

2. Stress Testing

// Aggressive limits to find breaking points
let config = ChaosConfig::aggressive();

// Run tracer on large program
// Identify resource bottlenecks

3. CI/CD Validation

# In CI pipeline
cargo test --features chaos-basic

# Run gentle chaos tests
CHAOS_MODE=gentle cargo test

4. Flaky Test Investigation

// Reproduce timing-sensitive bugs
let config = ChaosConfig::new()
    .with_cpu_limit(0.1)  // Slow down execution
    .with_signal_injection(true)  // Interrupt at random times
    .build();

Chaos vs. Fuzz Testing

Best Practice: Use both together!

# Tier 3 testing
make fuzz      # Input fuzzing
make chaos     # System chaos testing

Implementation Status (v0.4.1)

• ✅ ChaosConfig builder pattern
• ✅ Gentle/aggressive presets
• ✅ Property-based tests (7 tests)
• ✅ Cargo feature gates
• ⏳ CLI integration (planned Sprint 30)
• ⏳ Runtime chaos injection (planned Sprint 30)
• ⏳ Network chaos (planned Sprint 31)
• ⏳ Byzantine faults (planned Sprint 32)

Resources

• Chaos Engineering Book
• Netflix Chaos Engineering
• Aprender Chaos Patterns
• Loom Concurrency Testing

Example: Complete Chaos Test

use renacer::chaos::ChaosConfig;
use std::time::Duration;

#[test]
fn test_complete_chaos_scenario() {
    // Create aggressive chaos configuration
    let config = ChaosConfig::new()
        .with_memory_limit(20 * 1024 * 1024)  // 20MB
        .with_cpu_limit(0.25)                  // 25% CPU
        .with_timeout(Duration::from_secs(10))
        .with_signal_injection(true)
        .with_network_latency(Duration::from_millis(200))
        .with_packet_loss_rate(0.15)          // 15% loss
        .build();

    // Run tracer under chaos conditions
    let result = stress_test_tracer(config);

    // Verify graceful degradation
    match result {
        Ok(trace) => {
            // Success despite chaos!
            assert!(trace.syscalls.len() > 0);
        }
        Err(e) => {
            // Failed gracefully with meaningful error
            assert!(
                e.to_string().contains("timeout") ||
                e.to_string().contains("memory") ||
                e.to_string().contains("signal")
            );
        }
    }
}

Key Takeaways

1. Chaos reveals resilience - Finds bugs that normal testing misses
2. Progressive complexity - Start with basic, move to byzantine
3. Automated testing - Integrate into CI/CD pipeline
4. Graceful degradation - Systems should fail safely
5. Complementary to fuzzing - Use both for comprehensive testing