Regression Detection

Statistical hypothesis testing for detecting real performance regressions while filtering noise.

Key Innovation: No Magic Numbers

Traditional approaches use fixed percentage thresholds (e.g., "5% slowdown = regression"). This leads to high false positives because:

  1. Natural variance differs per project
  2. Some syscalls have high inherent variability (network I/O)
  3. Fixed thresholds don't adapt to baseline noise

Renacer's approach: Use statistical hypothesis testing (t-tests) with p-values that adapt to project-specific variance.

Problem: Fixed Thresholds Don't Work

Example: Natural Variance

Baseline read() times: [10ms, 12ms, 11ms, 13ms, 10ms]
Current read() times:  [11ms, 13ms, 12ms, 14ms, 11ms]

Fixed 5% threshold: 10ms → 12.6ms = +26% → FALSE POSITIVE!
Statistical test: p = 0.18 (not significant) → Correctly ignores

The difference is just natural variance, not a true regression.

Implementation

Statistical Comparison

use renacer::regression::{assess_regression, RegressionConfig};
use std::collections::HashMap;

// Baseline measurements (5 runs)
let mut baseline = HashMap::new();
baseline.insert("read".to_string(), vec![10.0, 12.0, 11.0, 13.0, 10.0]);
baseline.insert("mmap".to_string(), vec![5.0, 6.0, 5.0, 6.0, 5.0]);

// Current measurements (5 runs)
let mut current = HashMap::new();
current.insert("read".to_string(), vec![25.0, 27.0, 26.0, 28.0, 25.0]);  // REGRESSED!
current.insert("mmap".to_string(), vec![5.0, 6.0, 5.0, 6.0, 5.0]);      // Stable

let config = RegressionConfig::default();  // 95% confidence (p < 0.05)
let assessment = assess_regression(&baseline, &current, &config)?;

match assessment.verdict {
    RegressionVerdict::Regression { regressed_syscalls, .. } => {
        println!("⚠️ REGRESSION DETECTED");
        for syscall in regressed_syscalls {
            println!("  - {}", syscall);
        }
    }
    RegressionVerdict::NoRegression => {
        println!("✅ No regression detected");
    }
    _ => {}
}

Configuration Profiles

Default (Balanced)

RegressionConfig::default()
// - 95% confidence (p < 0.05)
// - 5 samples minimum
// - Noise filtering enabled (CV > 0.5)

Strict (Fewer False Positives)

RegressionConfig::strict()
// - 99% confidence (p < 0.01)
// - 10 samples minimum
// - Aggressive noise filtering (CV > 0.3)

Permissive (Catch Early)

RegressionConfig::permissive()
// - 90% confidence (p < 0.10)
// - 3 samples minimum
// - Relaxed noise filtering (CV > 1.0)

Noise Filtering: Delta Debugging

Based on Zeller (2002) Delta Debugging, Renacer filters out "noisy" syscalls before testing:

// Coefficient of Variation (CV) = std_dev / mean
//
// CV > threshold → noisy → filtered out
// CV ≤ threshold → stable → tested

Stable syscall:
  read: [10ms, 11ms, 10ms, 12ms, 10ms]
  CV = 0.08 (8% variance) → TESTED

Noisy syscall:
  socket: [5ms, 50ms, 3ms, 45ms, 2ms]  (network latency)
  CV = 0.96 (96% variance) → FILTERED

Why? High-variance syscalls cause false positives. By filtering them, we focus on stable, repeatable regressions.

Statistical Tests

Welch's T-Test

Renacer uses Welch's t-test (unequal variances) from the aprender library:

use aprender::stats::hypothesis::ttest_ind;

// Compare two distributions
let result = ttest_ind(baseline, current, false)?;

if result.pvalue < 0.05 {
    println!("Statistically significant difference (p = {:.4})", result.pvalue);
} else {
    println!("No significant difference (p = {:.4})", result.pvalue);
}

Why T-Tests?

  • Adaptive: Accounts for variance automatically
  • Robust: Works with small sample sizes (n ≥ 5)
  • Validated: 70+ years of statistical research

Real-World Example: decy Futex Regression

Baseline futex times:

[2ms, 3ms, 2ms, 3ms, 2ms]
Mean: 2.4ms, StdDev: 0.55ms, CV: 0.23

Current futex times (accidental async runtime):

[50ms, 52ms, 51ms, 53ms, 50ms]
Mean: 51.2ms, StdDev: 1.3ms, CV: 0.025

Statistical Test:

t-statistic: 89.3
p-value: < 0.001 (99.9% confidence)
Verdict: ⚠️ REGRESSION DETECTED

The difference is statistically significant - not just noise.

Report Format

let report = assessment.to_report_string();
println!("{}", report);

Output:

# Regression Detection Report

## Verdict: ⚠️ REGRESSION DETECTED

## Regressed Syscalls
- **futex**: 2.4ms → 51.2ms (+2033%, p < 0.001)
- **read**: 10.2ms → 15.8ms (+55%, p = 0.003)

## Stable Syscalls
- mmap: 5.5ms → 5.6ms (+2%, p = 0.89)
- write: 8.2ms → 8.3ms (+1%, p = 0.76)

## Filtered (Noisy) Syscalls
- socket: CV = 0.96 (high variance, unreliable)

## Statistical Tests
Total tests: 3
Significant: 2 (futex, read)
Not significant: 1 (mmap)
Filtered: 1 (socket)

## Recommendations
1. Investigate futex regression (+2033% is critical)
2. Profile read() operations (+55% is moderate)
3. Socket variance too high - consider multiple runs

CI/CD Integration

Build-Time Assertion

#[test]
fn test_no_performance_regression() {
    let baseline = load_golden_trace("golden.trace");
    let current = run_transpiler_and_trace("test.py");

    let config = RegressionConfig::default();
    let assessment = assess_regression(&baseline, &current, &config).unwrap();

    assert!(matches!(assessment.verdict, RegressionVerdict::NoRegression),
        "Performance regression detected:\n{}", assessment.to_report_string()
    );
}

GitHub Actions Example

- name: Performance Regression Check
  run: |
    cargo test test_no_performance_regression
  continue-on-error: false  # FAIL CI on regression

Implementation Statistics

  • Lines of Code: 1,285 lines (config, statistics, noise_filter, verdict)
  • Tests: 38/38 passing (100%)
  • Dependencies: aprender 0.7.1, trueno 0.7.0 (SIMD-optimized)
  • Zero Custom Implementations: Uses established libraries

API Reference

RegressionConfig

pub struct RegressionConfig {
    pub significance_level: f64,     // p-value threshold (default: 0.05)
    pub min_sample_size: usize,      // Minimum samples (default: 5)
    pub enable_noise_filtering: bool, // Filter high-CV syscalls (default: true)
    pub noise_threshold: f64,        // CV threshold (default: 0.5)
}

RegressionVerdict

pub enum RegressionVerdict {
    Regression {
        regressed_syscalls: Vec<String>,
        total_tests: usize,
        significant_tests: usize,
    },
    NoRegression,
    InsufficientData {
        reason: String,
    },
}

Peer-Reviewed Foundation

Based on:

  • Heger et al. (2013) "Automated Root Cause Isolation of Performance Regressions" (ICPE)

    • Finding: Fixed % thresholds have 40-60% false positive rate
    • Solution: Statistical hypothesis testing with p-values

  • Zeller (2002) "Isolating Cause-Effect Chains from Computer Programs" (FSE)

    • Delta Debugging for isolating relevant differences
    • Applied here for noise filtering

Toyota Way: Andon (Stop the Line)

Regressions trigger CI failures, stopping deployment:

⚠️ CI FAILED: Performance Regression Detected

Regressed: futex (+2033%)
Confidence: 99.9% (p < 0.001)

Action: Fix regression before merge.

Next Steps