Real-Time Anomaly Detection

Renacer provides real-time anomaly detection using sliding window statistics to identify unusual syscall behavior as your program runs.

TDD-Verified: All examples validated by tests/sprint20_realtime_anomaly_tests.rs

Overview

Real-time anomaly detection monitors syscall execution and alerts you immediately when unusual patterns occur:

• Live Monitoring: Detect anomalies as they happen (not post-hoc)
• Sliding Window Baselines: Per-syscall adaptive baselines using recent samples
• Severity Classification: Low (3-4σ), Medium (4-5σ), High (>5σ)
• SIMD-Accelerated: Trueno Vector operations for fast statistics
• Zero Overhead: No impact when disabled (opt-in via --anomaly-realtime)

Real-Time vs Post-Hoc Detection

Basic Usage

Enable Real-Time Detection

renacer --anomaly-realtime -T -- ./my-app

Tested by: test_realtime_anomaly_detects_slow_syscall

This enables real-time monitoring with:

• Default window size: 100 samples per syscall
• Default threshold: 3.0σ (standard deviations)
• Minimum samples: 10 per syscall before detection starts

Real-Time Alert Output

When an anomaly is detected, you'll see an immediate alert:

$ renacer --anomaly-realtime -T -- ./slow-app

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY) = 3
read(3, buf, 832) = 832
⚠️  ANOMALY: write took 5234 μs (4.2σ from baseline 102.3 μs) - 🟡 Medium
write(1, "processing...", 14) = 14
⚠️  ANOMALY: fsync took 8234 μs (6.3σ from baseline 123.4 μs) - 🔴 High
fsync(3) = 0
close(3) = 0

Tested by: test_realtime_anomaly_detects_slow_syscall

Each alert shows:

• Syscall name - Which syscall triggered the anomaly
• Duration - Actual time taken (μs)
• Z-score - How many standard deviations from baseline
• Baseline - Current mean for this syscall
• Severity - Visual indicator (🟢 Low, 🟡 Medium, 🔴 High)

Severity Classification

Anomalies are classified by their Z-score:

Tested by: test_anomaly_severity_classification

Example severity classification:

# 50ms delay on baseline 1ms syscall = ~50σ = High
⚠️  ANOMALY: write took 50234 μs (48.2σ from baseline 1023.4 μs) - 🔴 High

# 5ms delay on baseline 1ms syscall = ~5σ = High
⚠️  ANOMALY: read took 6123 μs (5.1σ from baseline 1123.4 μs) - 🔴 High

# 3.5ms delay on baseline 1ms syscall = ~3.5σ = Low
⚠️  ANOMALY: openat took 4234 μs (3.4σ from baseline 1023.4 μs) - 🟢 Low

Configuration

Custom Window Size

Control how many recent samples to keep per syscall:

renacer --anomaly-realtime --anomaly-window-size 50 -T -- ./app

Tested by: test_anomaly_window_size_configuration, test_anomaly_sliding_window_wraparound

• Default: 100 samples
• Smaller window (20-50): More sensitive to recent changes
• Larger window (200-500): More stable baseline, less noise

Window Size Trade-offs:

Minimum Samples

Anomaly detection requires at least 10 samples per syscall before alerts begin:

$ renacer --anomaly-realtime -T -e trace=write -- echo "test"
# No anomalies detected (only 1-2 write syscalls)

Tested by: test_anomaly_requires_minimum_samples

This prevents false alarms during application startup when baselines are unreliable.

Summary Report

After the trace completes, a summary report is displayed:

=== Real-Time Anomaly Detection Report ===
Total anomalies detected: 12

Severity Distribution:
  🔴 High (>5.0σ):   2 anomalies
  🟡 Medium (4-5σ): 5 anomalies
  🟢 Low (3-4σ):    5 anomalies

Top Anomalies (by Z-score):
  1. 🔴 fsync - 6.3σ (8234 μs, baseline: 123.4 ± 1287.2 μs)
  2. 🔴 write - 5.7σ (5234 μs, baseline: 102.3 ± 902.1 μs)
  3. 🟡 read - 4.8σ (2341 μs, baseline: 87.6 ± 468.9 μs)
  ... and 9 more

Tested by: test_realtime_anomaly_detects_slow_syscall

The report shows:

• Total count - Number of anomalies detected
• Severity distribution - Breakdown by Low/Medium/High
• Top anomalies - 10 most severe by Z-score
• Baseline statistics - Mean ± standard deviation

Integration with Other Features

With Statistics Mode (-c)

Combine real-time detection with summary statistics:

renacer -c --anomaly-realtime -T -- cargo build

Tested by: test_anomaly_realtime_with_statistics

Output includes:

1. Live alerts during execution (stderr)
2. Statistics table at the end (stderr)
3. Anomaly summary at the end (stderr)

With Filtering (-e)

Monitor anomalies only for specific syscalls:

renacer --anomaly-realtime -e trace=write -T -- ./app

Tested by: test_anomaly_realtime_with_filtering

This:

• Traces only filtered syscalls (e.g., write)
• Builds baselines only for filtered syscalls
• Detects anomalies only in filtered syscalls

Use case: Focus on I/O operations without noise from other syscalls.

With Multi-Process Tracing (-f)

Detect anomalies across all processes:

renacer -f --anomaly-realtime -T -- make -j8

Tested by: test_anomaly_realtime_with_multiprocess

Each process (parent + children) has:

• Independent baselines per syscall
• Separate anomaly detection (not shared across processes)

With JSON Output

Export anomalies to JSON for programmatic analysis:

renacer --anomaly-realtime -T --format json -- ./app > trace.json

Tested by: test_anomaly_json_export

JSON includes anomalies array in the output (if anomalies detected):

{
  "pid": 12345,
  "syscall": "write",
  "duration_us": 5234,
  "anomaly": {
    "z_score": 4.2,
    "baseline_mean": 102.3,
    "baseline_stddev": 902.1,
    "severity": "Medium"
  }
}

Edge Cases & Error Handling

Insufficient Samples

With too few syscalls, anomaly detection does not trigger:

$ renacer --anomaly-realtime -T -e trace=write -- echo "test"
# Output: No "ANOMALY" alerts (only 1 write syscall)

Tested by: test_anomaly_requires_minimum_samples

Minimum 10 samples required per syscall type before detection starts.

Zero Variance

When all samples are identical (stddev = 0), the system handles gracefully:

$ renacer --anomaly-realtime -T -- ./uniform-app
# No division-by-zero errors, no false anomalies

Tested by: test_anomaly_with_zero_variance

Implementation checks for stddev > 0.0 before calculating Z-score.

Sliding Window Wraparound

When sample count exceeds window size, old samples are removed:

$ renacer --anomaly-realtime --anomaly-window-size 50 -T -- ./many-syscalls
# Window maintains last 50 samples per syscall (FIFO)

Tested by: test_anomaly_sliding_window_wraparound

Memory usage stays constant (O(window_size × syscall_types)).

Backward Compatibility

Without --anomaly-realtime, no overhead occurs:

$ renacer -T -- ./app
# No anomaly detection, no performance impact

Tested by: test_backward_compatibility_without_anomaly_realtime

Ensures existing users see no behavior change.

How It Works

Sliding Window Statistics

For each syscall type (e.g., write, read):

1. Sample Collection: Last N durations stored (N = window size)
2. Statistics Update: After each syscall:
   • Calculate mean: μ = Σ(samples) / N
   • Calculate stddev: σ = √(Σ(x - μ)² / N)
   • Uses Trueno SIMD for fast computation
3. Anomaly Check: If |duration - μ| / σ > threshold:
   • Classify severity (Low/Medium/High)
   • Emit alert immediately
   • Store in summary

Per-Syscall Baselines

Each syscall type has independent baselines:

write:  μ = 102μs, σ = 45μs  (baseline from last 100 writes)
read:   μ = 523μs, σ = 234μs (baseline from last 100 reads)
fsync:  μ = 1234μs, σ = 567μs (baseline from last 100 fsyncs)

Why separate baselines?

• Different syscalls have different typical latencies
• fsync is naturally slower than write
• Comparing fsync to write baseline would always flag as anomaly

SIMD Acceleration

Uses Trueno library for fast statistics:

use trueno::Vector;

let v = Vector::from_slice(&samples);
let mean = v.mean().unwrap_or(0.0);     // SIMD-accelerated
let stddev = v.stddev().unwrap_or(0.0); // SIMD-accelerated

Performance: ~3-10x faster than naive loops for large windows.

Practical Examples

Example 1: Database Slow Query Detection

$ renacer --anomaly-realtime -e trace=file -T -- pg_bench

Output:

read(3, buf, 8192) = 8192
read(3, buf, 8192) = 8192
⚠️  ANOMALY: fsync took 15234 μs (8.2σ from baseline 1023.4 μs) - 🔴 High
fsync(3) = 0
read(3, buf, 8192) = 8192

Diagnosis: fsync taking 15ms instead of 1ms indicates:

• Disk I/O bottleneck
• WAL (Write-Ahead Log) blocking
• Consider: fsync=off for testing, SSD upgrade, or async I/O

Tested by: test_realtime_anomaly_detects_slow_syscall

Example 2: Network Latency Spikes

$ renacer --anomaly-realtime -e trace=network -T -- ./http_server

Output:

sendto(4, buf, 1024) = 1024
recvfrom(4, buf, 2048) = 512
⚠️  ANOMALY: recvfrom took 50234 μs (12.3σ from baseline 2023.4 μs) - 🔴 High
recvfrom(4, buf, 2048) = 512
sendto(4, buf, 1024) = 1024

Diagnosis: recvfrom taking 50ms instead of 2ms indicates:

• Network congestion
• Client-side delays
• Consider: Timeout adjustments, connection pooling

Example 3: CI/CD Pipeline Monitoring

$ renacer -f --anomaly-realtime -c -T -- make test

Use case: Detect slow build steps in multi-process builds:

• -f: Follow all child processes (parallel builds)
• --anomaly-realtime: Alert on slow I/O
• -c: Statistics summary at end
• -T: Timing data

Output identifies:

• Which subprocess had slow I/O
• Which syscalls were outliers
• Summary statistics for optimization

Troubleshooting

"No anomalies detected" but I know there are issues

Possible reasons:

1. Not enough samples:

   # Check: Run longer workload
   renacer --anomaly-realtime -T -- ./short-lived-app
   # Fix: Ensure app makes >10 syscalls per type
   

2. Threshold too high:

   # Default threshold is 3.0σ
   # For more sensitive detection, lower threshold (Sprint 19):
   renacer -c --stats-extended --anomaly-threshold 2.0 -- ./app
   

3. High variance baseline:

   • If syscall latency naturally varies (e.g., network I/O)
   • Anomalies may not exceed 3σ threshold
   • Check summary report for baseline stddev

Tested by: test_anomaly_requires_minimum_samples

Too many false positives

Solutions:

1. Increase threshold:

   # Use Sprint 19 post-hoc analysis with higher threshold
   renacer -c --stats-extended --anomaly-threshold 4.0 -- ./app
   

2. Increase window size:

   # More stable baselines = fewer alerts
   renacer --anomaly-realtime --anomaly-window-size 200 -T -- ./app
   

3. Filter specific syscalls:

   # Only monitor critical I/O operations
   renacer --anomaly-realtime -e trace=fsync,write -T -- ./app
   

Tested by: test_anomaly_window_size_configuration, test_anomaly_realtime_with_filtering

Anomaly detection not working with quick commands

Problem:

$ renacer --anomaly-realtime -T -- echo "test"
# No anomalies (only 1-2 syscalls)

Explanation: Need ≥10 samples per syscall type for reliable statistics.

Solutions:

1. Use longer-running workloads
2. Use post-hoc analysis instead (Sprint 19):

   renacer -c --stats-extended -- echo "test"
   

Tested by: test_anomaly_requires_minimum_samples

Comparison with Sprint 19 Post-Hoc Detection

When to Use Real-Time (Sprint 20)

✅ Use --anomaly-realtime when:

• Monitoring long-running applications
• Need immediate alerts (not post-analysis)
• Debugging live performance issues
• CI/CD pipeline monitoring

When to Use Post-Hoc (Sprint 19)

✅ Use --stats-extended when:

• Analyzing completed traces
• Need percentiles (P50, P75, P90, P95, P99)
• Short-lived commands (<10 syscalls per type)
• Historical analysis

Combined Approach

Use both for comprehensive analysis:

renacer -c --stats-extended --anomaly-realtime -T -- ./app

Tested by: test_anomaly_threshold_from_sprint19_still_works

This provides:

• Real-time alerts during execution
• Percentile analysis at the end
• Both Z-score methods (sliding window + global)

Performance

• Overhead: <1% when enabled (SIMD-accelerated statistics)
• Memory: O(window_size × syscall_types) - typically <1MB
• Speed: SIMD operations via Trueno (3-10x faster than naive loops)

Zero overhead when disabled (not enabled by default).

Summary

Real-time anomaly detection provides:

• ✅ Live monitoring with immediate alerts
• ✅ Sliding window baselines per syscall type
• ✅ Severity classification (Low/Medium/High with emojis)
• ✅ SIMD-accelerated statistics via Trueno
• ✅ Integration with filtering, multi-process, statistics, JSON
• ✅ Zero overhead when disabled (opt-in only)

All examples tested in: tests/sprint20_realtime_anomaly_tests.rs

Related

• Statistical Analysis - Post-hoc Z-score analysis (Sprint 19)
• Machine Learning - ML-based anomaly detection (Sprint 23)
• Filtering Syscalls - Focus detection with filters