Regression Detection - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# Regression Detection

Statistical hypothesis testing for detecting real performance regressions while filtering noise.

## Key Innovation: No Magic Numbers

Traditional approaches use fixed percentage thresholds (e.g., "5% slowdown = regression"). This leads to high false positives because:

1. Natural variance differs per project
2. Some syscalls have high inherent variability (network I/O)
3. Fixed thresholds don't adapt to baseline noise

Renacer's approach: Use statistical hypothesis testing (t-tests) with p-values that adapt to project-specific variance.

## Problem: Fixed Thresholds Don't Work

### Example: Natural Variance

    Baseline read() times: [10ms, 12ms, 11ms, 13ms, 10ms]
    Current read() times:  [11ms, 13ms, 12ms, 14ms, 11ms]

    Fixed 5% threshold: 10ms → 12.6ms = +26% → FALSE POSITIVE!
    Statistical test: p = 0.18 (not significant) → Correctly ignores

The difference is just natural variance, not a true regression.

## Implementation

### Statistical Comparison

    use renacer::regression::{assess_regression, RegressionConfig};
    use std::collections::HashMap;

    // Baseline measurements (5 runs)
    let mut baseline = HashMap::new();
    baseline.insert("read".to_string(), vec![10.0, 12.0, 11.0, 13.0, 10.0]);
    baseline.insert("mmap".to_string(), vec![5.0, 6.0, 5.0, 6.0, 5.0]);

    // Current measurements (5 runs)
    let mut current = HashMap::new();
    current.insert("read".to_string(), vec![25.0, 27.0, 26.0, 28.0, 25.0]);  // REGRESSED!
    current.insert("mmap".to_string(), vec![5.0, 6.0, 5.0, 6.0, 5.0]);      // Stable

    let config = RegressionConfig::default();  // 95% confidence (p < 0.05)
    let assessment = assess_regression(&baseline, &current, &config)?;

    match assessment.verdict {
        RegressionVerdict::Regression { regressed_syscalls, .. } => {
            println!("⚠️ REGRESSION DETECTED");
            for syscall in regressed_syscalls {
                println!("  - {}", syscall);
            }
        }
        RegressionVerdict::NoRegression => {
            println!("✅ No regression detected");
        }
        _ => {}
    }

## Configuration Profiles

### Default (Balanced)

    RegressionConfig::default()
    // - 95% confidence (p < 0.05)
    // - 5 samples minimum
    // - Noise filtering enabled (CV > 0.5)

### Strict (Fewer False Positives)

    RegressionConfig::strict()
    // - 99% confidence (p < 0.01)
    // - 10 samples minimum
    // - Aggressive noise filtering (CV > 0.3)

### Permissive (Catch Early)

    RegressionConfig::permissive()
    // - 90% confidence (p < 0.10)
    // - 3 samples minimum
    // - Relaxed noise filtering (CV > 1.0)

## Noise Filtering: Delta Debugging

Based on Zeller (2002) Delta Debugging, Renacer filters out "noisy" syscalls before testing:

    // Coefficient of Variation (CV) = std_dev / mean
    //
    // CV > threshold → noisy → filtered out
    // CV ≤ threshold → stable → tested

    Stable syscall:
      read: [10ms, 11ms, 10ms, 12ms, 10ms]
      CV = 0.08 (8% variance) → TESTED

    Noisy syscall:
      socket: [5ms, 50ms, 3ms, 45ms, 2ms]  (network latency)
      CV = 0.96 (96% variance) → FILTERED

Why? High-variance syscalls cause false positives. By filtering them, we focus on stable, repeatable regressions.

## Statistical Tests

### Welch's T-Test

Renacer uses Welch's t-test (unequal variances) from the aprender library:

    use aprender::stats::hypothesis::ttest_ind;

    // Compare two distributions
    let result = ttest_ind(baseline, current, false)?;

    if result.pvalue < 0.05 {
        println!("Statistically significant difference (p = {:.4})", result.pvalue);
    } else {
        println!("No significant difference (p = {:.4})", result.pvalue);
    }

### Why T-Tests?

- Adaptive: Accounts for variance automatically
- Robust: Works with small sample sizes (n ≥ 5)
- Validated: 70+ years of statistical research

## Real-World Example: decy Futex Regression

Baseline futex times:

    [2ms, 3ms, 2ms, 3ms, 2ms]
    Mean: 2.4ms, StdDev: 0.55ms, CV: 0.23

Current futex times (accidental async runtime):

    [50ms, 52ms, 51ms, 53ms, 50ms]
    Mean: 51.2ms, StdDev: 1.3ms, CV: 0.025

Statistical Test:

    t-statistic: 89.3
    p-value: < 0.001 (99.9% confidence)
    Verdict: ⚠️ REGRESSION DETECTED

The difference is statistically significant - not just noise.

## Report Format

    let report = assessment.to_report_string();
    println!("{}", report);

Output:

    # Regression Detection Report

    ## Verdict: ⚠️ REGRESSION DETECTED

    ## Regressed Syscalls
    - **futex**: 2.4ms → 51.2ms (+2033%, p < 0.001)
    - **read**: 10.2ms → 15.8ms (+55%, p = 0.003)

    ## Stable Syscalls
    - mmap: 5.5ms → 5.6ms (+2%, p = 0.89)
    - write: 8.2ms → 8.3ms (+1%, p = 0.76)

    ## Filtered (Noisy) Syscalls
    - socket: CV = 0.96 (high variance, unreliable)

    ## Statistical Tests
    Total tests: 3
    Significant: 2 (futex, read)
    Not significant: 1 (mmap)
    Filtered: 1 (socket)

    ## Recommendations
    1. Investigate futex regression (+2033% is critical)
    2. Profile read() operations (+55% is moderate)
    3. Socket variance too high - consider multiple runs

## CI/CD Integration

### Build-Time Assertion

    #[test]
    fn test_no_performance_regression() {
        let baseline = load_golden_trace("golden.trace");
        let current = run_transpiler_and_trace("test.py");

        let config = RegressionConfig::default();
        let assessment = assess_regression(&baseline, &current, &config).unwrap();

        assert!(matches!(assessment.verdict, RegressionVerdict::NoRegression),
            "Performance regression detected:\n{}", assessment.to_report_string()
        );
    }

### GitHub Actions Example

    - name: Performance Regression Check
      run: |
        cargo test test_no_performance_regression
      continue-on-error: false  # FAIL CI on regression

## Implementation Statistics

- Lines of Code: 1,285 lines (config, statistics, noise_filter, verdict)
- Tests: 38/38 passing (100%)
- Dependencies: aprender 0.7.1, trueno 0.7.0 (SIMD-optimized)
- Zero Custom Implementations: Uses established libraries

## API Reference

### RegressionConfig

    pub struct RegressionConfig {
        pub significance_level: f64,     // p-value threshold (default: 0.05)
        pub min_sample_size: usize,      // Minimum samples (default: 5)
        pub enable_noise_filtering: bool, // Filter high-CV syscalls (default: true)
        pub noise_threshold: f64,        // CV threshold (default: 0.5)
    }

### RegressionVerdict

    pub enum RegressionVerdict {
        Regression {
            regressed_syscalls: Vec<String>,
            total_tests: usize,
            significant_tests: usize,
        },
        NoRegression,
        InsufficientData {
            reason: String,
        },
    }

## Peer-Reviewed Foundation

Based on:

- Heger et al. (2013) "Automated Root Cause Isolation of Performance Regressions" (ICPE)

  - Finding: Fixed % thresholds have 40-60% false positive rate
  - Solution: Statistical hypothesis testing with p-values

- Zeller (2002) "Isolating Cause-Effect Chains from Computer Programs" (FSE)

  - Delta Debugging for isolating relevant differences
  - Applied here for noise filtering

## Toyota Way: Andon (Stop the Line)

Regressions trigger CI failures, stopping deployment:

    ⚠️ CI FAILED: Performance Regression Detected

    Regressed: futex (+2033%)
    Confidence: 99.9% (p < 0.001)

    Action: Fix regression before merge.

## Next Steps

- Combine with Time-Weighted Attribution for impact analysis
- Use Semantic Equivalence to validate fixes
- Integrate with Syscall Clustering for semantic grouping