# Regression Detection

Statistical hypothesis testing for detecting real performance regressions while filtering noise.

## Key Innovation: No Magic Numbers

Traditional approaches use **fixed percentage thresholds** (e.g., "5% slowdown = regression"). This leads to high false positives because:

1. Natural variance differs per project
2. Some syscalls have high inherent variability (network I/O)
3. Fixed thresholds don't adapt to baseline noise

**Renacer's approach**: Use **statistical hypothesis testing** (t-tests) with p-values that adapt to project-specific variance.

## Problem: Fixed Thresholds Don't Work

### Example: Natural Variance

```text
Baseline read() times: [10ms, 12ms, 11ms, 13ms, 10ms]
Current read() times:  [11ms, 13ms, 12ms, 14ms, 11ms]

Fixed 5% threshold: 10ms → 12.6ms = +26% → FALSE POSITIVE!
Statistical test: p = 0.18 (not significant) → Correctly ignores
```

The difference is just natural variance, not a true regression.

## Implementation

### Statistical Comparison

```rust
use renacer::regression::{assess_regression, RegressionConfig};
use std::collections::HashMap;

// Baseline measurements (5 runs)
let mut baseline = HashMap::new();
baseline.insert("read".to_string(), vec![10.0, 12.0, 11.0, 13.0, 10.0]);
baseline.insert("mmap".to_string(), vec![5.0, 6.0, 5.0, 6.0, 5.0]);

// Current measurements (5 runs)
let mut current = HashMap::new();
current.insert("read".to_string(), vec![25.0, 27.0, 26.0, 28.0, 25.0]);  // REGRESSED!
current.insert("mmap".to_string(), vec![5.0, 6.0, 5.0, 6.0, 5.0]);      // Stable

let config = RegressionConfig::default();  // 95% confidence (p < 0.05)
let assessment = assess_regression(&baseline, &current, &config)?;

match assessment.verdict {
    RegressionVerdict::Regression { regressed_syscalls, .. } => {
        println!("⚠️ REGRESSION DETECTED");
        for syscall in regressed_syscalls {
            println!("  - {}", syscall);
        }
    }
    RegressionVerdict::NoRegression => {
        println!("✅ No regression detected");
    }
    _ => {}
}
```

## Configuration Profiles

### Default (Balanced)
```rust
RegressionConfig::default()
// - 95% confidence (p < 0.05)
// - 5 samples minimum
// - Noise filtering enabled (CV > 0.5)
```

### Strict (Fewer False Positives)
```rust
RegressionConfig::strict()
// - 99% confidence (p < 0.01)
// - 10 samples minimum
// - Aggressive noise filtering (CV > 0.3)
```

### Permissive (Catch Early)
```rust
RegressionConfig::permissive()
// - 90% confidence (p < 0.10)
// - 3 samples minimum
// - Relaxed noise filtering (CV > 1.0)
```

## Noise Filtering: Delta Debugging

Based on **Zeller (2002)** Delta Debugging, Renacer filters out "noisy" syscalls before testing:

```rust
// Coefficient of Variation (CV) = std_dev / mean
//
// CV > threshold → noisy → filtered out
// CV ≤ threshold → stable → tested

Stable syscall:
  read: [10ms, 11ms, 10ms, 12ms, 10ms]
  CV = 0.08 (8% variance) → TESTED

Noisy syscall:
  socket: [5ms, 50ms, 3ms, 45ms, 2ms]  (network latency)
  CV = 0.96 (96% variance) → FILTERED
```

**Why?** High-variance syscalls cause false positives. By filtering them, we focus on **stable, repeatable** regressions.

## Statistical Tests

### Welch's T-Test

Renacer uses **Welch's t-test** (unequal variances) from the aprender library:

```rust
use aprender::stats::hypothesis::ttest_ind;

// Compare two distributions
let result = ttest_ind(baseline, current, false)?;

if result.pvalue < 0.05 {
    println!("Statistically significant difference (p = {:.4})", result.pvalue);
} else {
    println!("No significant difference (p = {:.4})", result.pvalue);
}
```

### Why T-Tests?

- **Adaptive**: Accounts for variance automatically
- **Robust**: Works with small sample sizes (n ≥ 5)
- **Validated**: 70+ years of statistical research

## Real-World Example: decy Futex Regression

**Baseline futex times**:
```text
[2ms, 3ms, 2ms, 3ms, 2ms]
Mean: 2.4ms, StdDev: 0.55ms, CV: 0.23
```

**Current futex times** (accidental async runtime):
```text
[50ms, 52ms, 51ms, 53ms, 50ms]
Mean: 51.2ms, StdDev: 1.3ms, CV: 0.025
```

**Statistical Test**:
```text
t-statistic: 89.3
p-value: < 0.001 (99.9% confidence)
Verdict: ⚠️ REGRESSION DETECTED
```

The difference is **statistically significant** - not just noise.

## Report Format

```rust
let report = assessment.to_report_string();
println!("{}", report);
```

**Output**:
```text
# Regression Detection Report

## Verdict: ⚠️ REGRESSION DETECTED

## Regressed Syscalls
- **futex**: 2.4ms → 51.2ms (+2033%, p < 0.001)
- **read**: 10.2ms → 15.8ms (+55%, p = 0.003)

## Stable Syscalls
- mmap: 5.5ms → 5.6ms (+2%, p = 0.89)
- write: 8.2ms → 8.3ms (+1%, p = 0.76)

## Filtered (Noisy) Syscalls
- socket: CV = 0.96 (high variance, unreliable)

## Statistical Tests
Total tests: 3
Significant: 2 (futex, read)
Not significant: 1 (mmap)
Filtered: 1 (socket)

## Recommendations
1. Investigate futex regression (+2033% is critical)
2. Profile read() operations (+55% is moderate)
3. Socket variance too high - consider multiple runs
```

## CI/CD Integration

### Build-Time Assertion

```rust
#[test]
fn test_no_performance_regression() {
    let baseline = load_golden_trace("golden.trace");
    let current = run_transpiler_and_trace("test.py");

    let config = RegressionConfig::default();
    let assessment = assess_regression(&baseline, &current, &config).unwrap();

    assert!(matches!(assessment.verdict, RegressionVerdict::NoRegression),
        "Performance regression detected:\n{}", assessment.to_report_string()
    );
}
```

### GitHub Actions Example

```yaml
- name: Performance Regression Check
  run: |
    cargo test test_no_performance_regression
  continue-on-error: false  # FAIL CI on regression
```

## Implementation Statistics

- **Lines of Code**: 1,285 lines (config, statistics, noise_filter, verdict)
- **Tests**: 38/38 passing (100%)
- **Dependencies**: aprender 0.7.1, trueno 0.7.0 (SIMD-optimized)
- **Zero Custom Implementations**: Uses established libraries

## API Reference

### RegressionConfig

```rust
pub struct RegressionConfig {
    pub significance_level: f64,     // p-value threshold (default: 0.05)
    pub min_sample_size: usize,      // Minimum samples (default: 5)
    pub enable_noise_filtering: bool, // Filter high-CV syscalls (default: true)
    pub noise_threshold: f64,        // CV threshold (default: 0.5)
}
```

### RegressionVerdict

```rust
pub enum RegressionVerdict {
    Regression {
        regressed_syscalls: Vec<String>,
        total_tests: usize,
        significant_tests: usize,
    },
    NoRegression,
    InsufficientData {
        reason: String,
    },
}
```

## Peer-Reviewed Foundation

Based on:

- **Heger et al. (2013)** "Automated Root Cause Isolation of Performance Regressions" (ICPE)
  - Finding: Fixed % thresholds have 40-60% false positive rate
  - Solution: Statistical hypothesis testing with p-values

- **Zeller (2002)** "Isolating Cause-Effect Chains from Computer Programs" (FSE)
  - Delta Debugging for isolating relevant differences
  - Applied here for noise filtering

## Toyota Way: Andon (Stop the Line)

Regressions trigger CI failures, stopping deployment:

```text
⚠️ CI FAILED: Performance Regression Detected

Regressed: futex (+2033%)
Confidence: 99.9% (p < 0.001)

Action: Fix regression before merge.
```

## Next Steps

- Combine with [Time-Weighted Attribution](./time-attribution.md) for impact analysis
- Use [Semantic Equivalence](./semantic-equivalence.md) to validate fixes
- Integrate with [Syscall Clustering](./syscall-clustering.md) for semantic grouping
