Statistical Analysis - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# Statistical Analysis

Renacer provides SIMD-accelerated statistical analysis of syscall performance using the Trueno library, enabling deep insights into latency distributions and anomaly detection.

> TDD-Verified: All examples validated by `tests/sprint19_enhanced_stats_tests.rs`

## Overview

Enhanced statistical analysis provides comprehensive latency metrics beyond basic averages:

- Percentile Analysis: P50, P75, P90, P95, P99 latency distributions
- Descriptive Statistics: Mean, standard deviation, min, max
- Post-Hoc Anomaly Detection: Z-score based outlier identification
- SIMD-Accelerated: Trueno Vector operations for 3-10x faster computation
- Zero Overhead: No impact when disabled (opt-in via `--stats-extended`)

### Post-Hoc vs Real-Time Analysis

Feature | Post-Hoc (Sprint 19) | Real-Time (Sprint 20)
--- | --- | ---
Detection | After trace completes | Live during execution
Baseline | All samples (global mean) | Sliding window (last N samples)
Use Case | Historical analysis, percentiles | Monitor long-running apps
Overhead | None (post-processing) | Minimal (<1%)
Flag | `--stats-extended` | `--anomaly-realtime`

## Basic Usage

### Enable Extended Statistics

    renacer -c --stats-extended -T -- cargo build

Tested by: `test_stats_extended_calculates_percentiles`

This enables:

- Percentile calculations: P50, P75, P90, P95, P99
- Descriptive statistics: Mean, StdDev, Min, Max
- Anomaly detection: Z-score based outliers (threshold: 3.0σ)
- SIMD acceleration: Trueno Vector operations

Note: Requires `-T` flag for timing data. Without `-T`, only call counts are shown.

### Extended Statistics Output

    $ renacer -c --stats-extended -T -- cargo test

    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     65.43    0.142301        4234        42         0 read
     18.92    0.041234        2062        20         0 write
     10.23    0.022301         892        25         0 openat
      3.21    0.007001         700        10         0 close
      2.21    0.004812         481        10         0 mmap
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.217649                   107         0 total

    === Extended Statistics (SIMD-accelerated via Trueno) ===

    read (42 calls):
      Mean:         4234.50 μs
      Std Dev:      1234.67 μs
      Min:          2123.00 μs
      Max:          9234.00 μs
      Median (P50): 3890.00 μs
      P75:          5123.00 μs
      P90:          6234.00 μs
      P95:          7123.00 μs
      P99:          8934.00 μs

    write (20 calls):
      Mean:         2062.00 μs
      Std Dev:      823.45 μs
      Min:          1023.00 μs
      Max:          4234.00 μs
      Median (P50): 1980.00 μs
      P75:          2456.00 μs
      P90:          3123.00 μs
      P95:          3678.00 μs
      P99:          4123.00 μs

    === Post-Hoc Anomaly Detection (threshold: 3.0σ) ===
    2 anomalies detected:
      - read: 9234.00 μs (3.8σ above mean)
      - write: 4234.00 μs (3.2σ above mean)

Tested by: `test_stats_extended_calculates_percentiles`, `test_stats_extended_shows_min_max`

Each syscall type shows:

- Mean - Average duration
- Std Dev - Standard deviation (variance measure)
- Min/Max - Fastest and slowest execution
- Percentiles - Distribution breakdown (P50=median, P95=95th percentile, etc.)

## Percentile Interpretation

Percentiles show the latency distribution:

Percentile | Meaning
--- | ---
P50 (Median) | 50% of calls are faster than this value
P75 | 75% of calls are faster than this value
P90 | 90% of calls are faster than this value
P95 | 95% of calls are faster than this value
P99 | 99% of calls are faster than this value (tail latency)

Example Interpretation:

    read (42 calls):
      Median (P50): 3890.00 μs   # Typical latency
      P95:          7123.00 μs   # 95% complete under 7ms
      P99:          8934.00 μs   # Worst 1% take ~9ms (tail latency)

- P50 (3.9ms): Most reads complete in ~4ms
- P95 (7.1ms): 5% of reads are slower (potential outliers)
- P99 (8.9ms): 1% of reads are very slow (investigate these)

Tested by: `test_stats_extended_calculates_percentiles`

## Post-Hoc Anomaly Detection

Anomalies are identified using Z-score analysis:

    $ renacer -c --stats-extended -T -- ./slow-app

    === Post-Hoc Anomaly Detection (threshold: 3.0σ) ===
    3 anomalies detected:
      - fsync: 15234.00 μs (6.3σ above mean)
      - write: 5234.00 μs (4.2σ above mean)
      - read: 2341.00 μs (3.5σ above mean)

Tested by: `test_anomaly_detection_slow_syscall`

Z-Score Meaning:

- 3.0σ-4.0σ: Noticeable outlier
- 4.0σ-5.0σ: Significant outlier
- >5.0σ: Extreme outlier (investigate!)

### Custom Anomaly Threshold

Adjust sensitivity with `--anomaly-threshold`:

    renacer -c --stats-extended --anomaly-threshold 2.5 -T -- ./app

Tested by: `test_anomaly_threshold_configuration`

- Default: 3.0σ (captures significant outliers)
- Lower (2.0-2.5σ): More sensitive (more alerts)
- Higher (4.0-5.0σ): Less sensitive (only extreme outliers)

Trade-offs:

Threshold | Sensitivity | False Positives | Use Case
--- | --- | --- | ---
2.0σ | Very high | Many | Aggressive optimization
2.5σ | High | Some | Development debugging
3.0σ | Moderate | Few | Production analysis (default)
4.0σ | Low | Rare | Critical bottlenecks only

## Integration with Other Features

### With Timing Mode (-T)

Required for duration-based statistics:

    renacer -c --stats-extended -T -- ./app

Tested by: `test_stats_extended_with_timing`

Without `-T`, only call counts are shown:

    $ renacer -c --stats-extended -- ./app
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    # No timing data available for percentiles

Tested by: `test_stats_extended_no_timing_data`

### With Filtering (-e)

Analyze only specific syscalls:

    renacer -c --stats-extended -e trace=write,read -T -- ./app

Tested by: `test_stats_extended_with_filtering`

Output shows extended statistics only for filtered syscalls (write, read).

Use case: Focus analysis on I/O operations without noise from other syscalls.

### With Multi-Process Tracing (-f)

Aggregate statistics across all processes:

    renacer -f -c --stats-extended -T -- make -j8

Tested by: `test_stats_extended_with_multiprocess`

Statistics combine data from:

- Parent process
- All child processes (fork, vfork, clone)

Use case: Analyze parallel build performance, identify slowest subprocess.

### With JSON Output

Statistics summary goes to stderr, trace goes to stdout:

    renacer -c --stats-extended -T --format json -- ./app > trace.json 2> stats.txt

Tested by: `test_stats_extended_json_output`

Output split:

- stdout: JSON trace data
- stderr: Extended statistics summary (human-readable)

### With CSV Output

Statistics summary goes to stderr, CSV goes to stdout:

    renacer -c --stats-extended -T --format csv -- ./app > trace.csv 2> stats.txt

Tested by: `test_stats_extended_csv_output`

Output split:

- stdout: CSV trace data
- stderr: Extended statistics summary (human-readable)

## Edge Cases & Error Handling

### Single Syscall (No Variance)

With only one data point:

    $ renacer -c --stats-extended -T -- echo "test"

    write (1 calls):
      Mean:         1234.00 μs
      Std Dev:      0.00 μs       # No variance with single sample
      Min:          1234.00 μs
      Max:          1234.00 μs
      Median (P50): 1234.00 μs
      P75:          1234.00 μs
      P90:          1234.00 μs
      P95:          1234.00 μs
      P99:          1234.00 μs    # All percentiles equal

    === Post-Hoc Anomaly Detection (threshold: 3.0σ) ===
    0 anomalies detected (stddev = 0, cannot compute Z-score)

Tested by: `test_stats_extended_single_syscall`

No anomalies can be detected when stddev = 0 (division by zero avoided).

### No Timing Data

Without `-T` flag:

    $ renacer -c --stats-extended -- ./app

    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     # Standard statistics table (counts only)

    === Extended Statistics ===
    # No duration data available (use -T flag for timing)

Tested by: `test_stats_extended_no_timing_data`

Percentile analysis requires duration data (`-T` flag).

### Large Datasets

SIMD acceleration handles large traces efficiently:

    $ renacer -c --stats-extended -T -- ./high-volume-app
    # 1000+ syscalls per type processed efficiently via Trueno

Tested by: `test_stats_extended_large_dataset`

Performance: Trueno SIMD operations are 3-10x faster than naive loops for large datasets.

### Backward Compatibility

Without `--stats-extended`, no overhead occurs:

    $ renacer -c -T -- ./app
    # Standard statistics table only (no extended stats)

Tested by: `test_backward_compatibility_without_stats_extended`

Ensures existing users see no behavior change.

## How It Works

### SIMD-Accelerated Statistics

Uses Trueno library for high-performance vector operations:

    use trueno::Vector;

    // Convert durations to vector
    let durations: Vec<f32> = stats.durations.iter().map(|&d| d as f32).collect();
    let v = Vector::from_slice(&durations);

    // SIMD-accelerated computations
    let mean = v.mean().unwrap_or(0.0);     // Vectorized mean
    let stddev = v.stddev().unwrap_or(0.0); // Vectorized standard deviation
    let min = v.min().unwrap_or(0.0);       // Vectorized min
    let max = v.max().unwrap_or(0.0);       // Vectorized max

Performance Benefit: 3-10x faster than scalar loops for large datasets.

### Percentile Calculation

Percentiles calculated via interpolation on sorted data:

1. Sort durations: `[100, 150, 200, 250, 300]`
2. Calculate index: For P90: `0.90 * (5-1) = 3.6`
3. Interpolate: Between index 3 (250) and 4 (300): `250 + 0.6*(300-250) = 280`
4. Result: P90 = 280μs

Implementation:

    fn calculate_percentile(sorted_data: &[f32], percentile: f32) -> f32 {
        let index = (percentile / 100.0) * (sorted_data.len() - 1) as f32;
        let lower = index.floor() as usize;
        let upper = index.ceil() as usize;

        if lower == upper {
            sorted_data[lower]
        } else {
            let weight = index - lower as f32;
            sorted_data[lower] * (1.0 - weight) + sorted_data[upper] * weight
        }
    }

### Z-Score Anomaly Detection

Anomalies identified using statistical Z-score:

    Z-score = (duration - mean) / stddev

Example:

    write syscall: duration = 5234μs
    Baseline: mean = 1023μs, stddev = 987μs

    Z-score = (5234 - 1023) / 987 = 4.26σ

    Result: 4.26σ > 3.0σ threshold → ANOMALY

Classification:

- Z > 3.0σ: Anomaly detected
- Severity based on magnitude (3-4σ: Low, 4-5σ: Medium, >5σ: High)

## Practical Examples

### Example 1: Identifying Tail Latency

    $ renacer -c --stats-extended -T -e trace=read -- ./database-app

    read (1000 calls):
      Mean:         1234.00 μs
      Std Dev:      456.00 μs
      Min:          823.00 μs
      Max:          8234.00 μs
      Median (P50): 1123.00 μs   # Typical read: ~1ms
      P95:          2123.00 μs   # 95% under 2ms ✅
      P99:          5234.00 μs   # Worst 1% take 5ms+ ⚠️

    === Post-Hoc Anomaly Detection ===
    10 anomalies detected (P99+ outliers)

Diagnosis:

- P50-P95 are reasonable (~1-2ms)
- P99 jumps to 5ms+ - tail latency issue
- Anomalies at P99 - disk I/O spikes or cache misses

Action: Investigate P99 reads (likely disk-bound, consider caching).

Tested by: `test_stats_extended_calculates_percentiles`

### Example 2: Comparing Before/After Optimization

Before optimization:

    $ renacer -c --stats-extended -T -- ./app-v1

    write (500 calls):
      Median (P50): 2345.00 μs
      P95:          5678.00 μs
      P99:          8234.00 μs

After optimization:

    $ renacer -c --stats-extended -T -- ./app-v2

    write (500 calls):
      Median (P50): 1234.00 μs   # 47% faster ✅
      P95:          2345.00 μs   # 59% faster ✅
      P99:          3456.00 μs   # 58% faster ✅

Result: Optimization improved both typical (P50) and tail (P99) latency.

Tested by: `test_stats_extended_large_dataset`

### Example 3: CI/CD Performance Regression Detection

    $ renacer -f -c --stats-extended -T -- make test

    # Baseline (commit A): P95 = 2.3ms
    # Current (commit B): P95 = 5.6ms ⚠️ REGRESSION

    === Post-Hoc Anomaly Detection ===
    25 anomalies detected (up from 5 in baseline)

Diagnosis: Recent commit introduced performance regression.

Action: Bisect commits to find regression source.

Tested by: `test_stats_extended_with_multiprocess`

## Troubleshooting

### "No duration data available"

Problem:

    $ renacer -c --stats-extended -- ./app
    # No percentiles shown

Solution: Add `-T` flag for timing data:

    $ renacer -c --stats-extended -T -- ./app

Tested by: `test_stats_extended_no_timing_data`

### "0 anomalies detected" but I see slow syscalls

Possible reasons:

1. High variance baseline:

   - If syscalls naturally vary (e.g., network I/O), stddev is high
   - Slow syscalls may not exceed 3σ threshold
   - Solution: Lower threshold: `--anomaly-threshold 2.0`

2. Insufficient samples:

   - With <10 samples, anomaly detection may be unreliable
   - Solution: Run longer workload or use real-time detection (Sprint 20)

3. Outliers within normal distribution:

   - P99 may be slow but still within 3σ
   - Solution: Check percentiles (P95, P99) manually

Tested by: `test_anomaly_detection_slow_syscall`

### Too many false positives

Problem:

    === Post-Hoc Anomaly Detection ===
    50 anomalies detected (many false positives)

Solutions:

1. Increase threshold:

       renacer -c --stats-extended --anomaly-threshold 4.0 -T -- ./app

2. Filter noisy syscalls:

       # Only analyze critical I/O operations
       renacer -c --stats-extended -e trace=fsync,write -T -- ./app

3. Use percentiles instead:

   - Focus on P95/P99 values rather than anomaly count
   - Percentiles show distribution without binary anomaly classification

Tested by: `test_anomaly_threshold_configuration`, `test_stats_extended_with_filtering`

### Extended stats not showing with JSON/CSV

Expected behavior: Extended stats go to stderr, traces go to stdout.

Solution: Capture both streams:

    renacer -c --stats-extended -T --format json -- ./app > trace.json 2> stats.txt

Tested by: `test_stats_extended_json_output`, `test_stats_extended_csv_output`

## Comparison with Real-Time Detection

### When to Use Post-Hoc (Sprint 19)

✅ Use `--stats-extended` when:

- Analyzing completed traces
- Need percentile distributions (P50, P75, P90, P95, P99)
- Short-lived commands
- Historical performance analysis
- Regression testing (compare before/after)

### When to Use Real-Time (Sprint 20)

✅ Use `--anomaly-realtime` when:

- Monitoring long-running applications
- Need immediate alerts during execution
- Debugging live performance issues
- CI/CD pipeline monitoring

### Combined Approach

Use both for comprehensive analysis:

    renacer -c --stats-extended --anomaly-realtime -T -- ./app

Provides:

- Real-time alerts during execution (sliding window)
- Post-hoc analysis at the end (global statistics)
- Percentile distributions for historical comparison
- Both Z-score methods (sliding window + global)

## Performance

- Overhead: None (post-processing after trace completes)
- Memory: O(n) where n = total syscalls (stores all durations)
- Speed: 3-10x faster via Trueno SIMD compared to scalar loops
- Large datasets: Handles 1000+ syscalls per type efficiently

Zero overhead when disabled (not enabled by default).

## Summary

Statistical analysis provides:

- ✅ Percentile distributions (P50, P75, P90, P95, P99)
- ✅ Descriptive statistics (mean, stddev, min, max)
- ✅ Post-hoc anomaly detection via Z-score
- ✅ SIMD-accelerated via Trueno (3-10x faster)
- ✅ Integration with filtering, multi-process, JSON, CSV
- ✅ Zero overhead when disabled (opt-in only)

All examples tested in: `tests/sprint19_enhanced_stats_tests.rs`

## Related

- Real-Time Anomaly Detection - Live monitoring (Sprint 20)
- Machine Learning - ML-based anomaly detection (Sprint 23)
- Basic Tracing - Core syscall tracing