Statistics Mode - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# Statistics Mode

When you need to understand overall system call behavior rather than individual calls, statistics mode (`-c` flag) provides aggregate analysis. Instead of thousands of lines of syscall traces, you get a concise summary of what happened.

## What is Statistics Mode?

Statistics mode counts and times syscalls, then displays a summary table instead of per-syscall output.

### Basic Usage

    renacer -c -- command

### Example

Without statistics:

    $ renacer -- ls
    openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
    fstat(3, {...}) = 0
    mmap(NULL, 163352, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9a2c000000
    # ... 200+ more lines ...

With statistics:

    $ renacer -c -- ls
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time    Min Time    Max Time
    openat           5        0         2.345ms       0.469ms     0.123ms     1.234ms
    fstat            8        0         0.891ms       0.111ms     0.089ms     0.156ms
    read             3        0         1.234ms       0.411ms     0.234ms     0.678ms
    mmap             12       0         3.456ms       0.288ms     0.145ms     0.567ms
    write            2        0         0.567ms       0.284ms     0.234ms     0.334ms
    close            5        0         0.234ms       0.047ms     0.034ms     0.067ms

Result: 200+ lines reduced to 6 summary rows.

## Why Aggregate Analysis?

### 1. Performance Profiling

Question: Which syscalls are slowing down my application?

    $ renacer -c -- ./slow-app
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    fsync            1247     0         5.678s        4.553ms     # ⚠️ Slow!
    openat           1247     0         2.345s        1.881ms
    write            3741     0         1.234s        0.330ms

Answer: `fsync` is taking 5.6 seconds total (45% of execution time).

### 2. Error Analysis

Question: How many errors occurred?

    $ renacer -c -- ./buggy-app
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    openat           1500     247       1.234s        0.823ms     # 247 errors!
    read             1253     0         0.567s        0.453ms

Answer: `openat` failed 247 times (16% failure rate).

### 3. Syscall Frequency

Question: Which syscalls are called most?

    $ renacer -c -- cargo build
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    read             45678    0         12.345s       0.270ms     # Most frequent
    write            23456    0         8.901s        0.379ms
    openat           12345    0         5.678s        0.460ms

Answer: `read` is called 45,678 times during build.

## Understanding the Output

### Column Descriptions

Column | Description | Example | Interpretation
--- | --- | --- | ---
Syscall | Syscall name | `openat` | Which syscall
Calls | Total number of calls | `1247` | Frequency
Errors | Number of failed calls | `247` | Failure count
Total Time | Cumulative time | `5.678s` | Total time spent
Avg Time | Average per call | `4.553ms` | Typical duration
Min Time | Fastest call | `0.123ms` | Best case
Max Time | Slowest call | `23.456ms` | Worst case

### Sorting

By default, output is sorted by Total Time (descending) - showing most time-consuming syscalls first.

Example:

    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    fsync            100      0         5.678s        56.78ms     # #1: Most total time
    read             5000     0         3.456s        0.691ms     # #2
    write            3000     0         2.345s        0.782ms     # #3

## Enhanced Statistics (Sprint 19+)

Renacer provides advanced statistical analysis beyond basic averages.

### Percentile Analysis

    $ renacer -c -- ./app
    System Call Summary (Enhanced):
    ================================
    Syscall          Calls    Total Time    Avg      Min      p50      p90      p99      Max
    read             5000     3.456s        0.691ms  0.123ms  0.567ms  1.234ms  2.345ms  5.678ms
    write            3000     2.345s        0.782ms  0.234ms  0.678ms  1.456ms  3.456ms  8.901ms
    fsync            100      5.678s        56.78ms  12.34ms  45.67ms  89.01ms  123.45ms 234.56ms

Percentiles explained:

- p50 (median): 50% of calls faster than this
- p90: 90% of calls faster than this (90th percentile)
- p99: 99% of calls faster than this (outlier detection)

### Interpreting Percentiles

Example: `read` syscall

    Avg: 0.691ms  p50: 0.567ms  p90: 1.234ms  p99: 2.345ms  Max: 5.678ms

Analysis:

- p50 < Avg: Distribution is right-skewed (few slow outliers pull average up)
- p90 = 2x p50: 10% of reads take 2x longer than median
- p99 = 4x p50: 1% of reads are extremely slow
- Max >> p99: One outlier took 5.6ms (10x median)

Conclusion: Most reads are fast (~0.5ms), but occasional slow reads (p99) indicate I/O contention or disk latency spikes.

### SIMD-Accelerated Percentiles

Renacer uses SIMD instructions (AVX2/NEON) for fast percentile calculation on large datasets:

    $ renacer -c -- stress-test  # 1M+ syscalls
    # Percentiles computed using SIMD in <100ms

This makes statistics mode practical even for high-frequency tracing.

## Combining with Filtering

Statistics mode works seamlessly with syscall filtering.

### File Operations Only

    $ renacer -c -e 'trace=file' -- ./app
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    openat           1247     23        2.345s        1.881ms
    read             3741     0         1.234s        0.330ms
    write            1867     0         0.891s        0.477ms
    close            1224     0         0.123s        0.101ms

### Network Operations Only

    $ renacer -c -e 'trace=network' -- curl https://example.com
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    recvfrom         234      0         1.234s        5.274ms
    sendto           178      0         0.567s        3.185ms
    connect          3        1         0.234s        78.0ms      # 1 error!
    socket           3        0         0.012s        4.0ms

Insight: `connect` failed once (probably timeout/refused connection).

### Specific Syscalls

    $ renacer -c -e 'trace=read,write' -- cat large-file.txt
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    read             1024     0         2.345s        2.290ms
    write            1024     0         1.234s        1.205ms

## Real-World Performance Analysis

### Scenario 1: Slow Startup

Problem: Application takes 10 seconds to start.

    $ renacer -c -- ./slow-startup
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    openat           5678     0         7.890s        1.390ms     # 79% of startup time!
    fstat            5678     0         1.234s        0.217ms
    read             17034    0         0.891s        0.052ms

Analysis:

- `openat` dominates with 7.9s (79% of time)
- Called 5,678 times
- Average 1.4ms per call (seems slow for file open)

Investigation:

    $ renacer -e 'trace=openat' -- ./slow-startup
    openat(AT_FDCWD, "/usr/share/icons/hicolor/16x16/apps/icon001.png", O_RDONLY) = 3
    openat(AT_FDCWD, "/usr/share/icons/hicolor/16x16/apps/icon002.png", O_RDONLY) = 3
    # ... 5676 more icons ...

Problem: Loading 5,678 icons individually during startup.

Solution: Lazy-load icons or bundle them.

### Scenario 2: Network Latency

Problem: API client seems slow.

    $ renacer -c -e 'trace=network' -- ./api-client
    System Call Summary (Enhanced):
    ================================
    Syscall          Calls    Avg      p50      p90      p99      Max
    recvfrom         500      34.5ms   12.3ms   89.0ms   234.5ms  567.8ms
    sendto           500      2.3ms    1.2ms    4.5ms    12.3ms   23.4ms

Analysis:

- p50 (12.3ms): Typical network round-trip is fast
- p90 (89.0ms): 10% of requests take 7x longer
- p99 (234.5ms): 1% take 20x longer
- Max (567.8ms): One request took half a second

Conclusion: Network latency is highly variable. Possible causes:

- Server under load (slow p90/p99)
- Network congestion
- DNS resolution delays

Solution: Add retry logic with exponential backoff for slow requests.

### Scenario 3: File I/O Bottleneck

Problem: Data processing is slower than expected.

    $ renacer -c -e 'trace=read,write' -- ./data-processor input.csv
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time
    read             100000   0         45.678s       0.457ms
    write            100000   0         12.345s       0.123ms

Analysis:

- Reading 100K times taking 45 seconds (73% of time)
- Average 0.457ms per read (seems slow for in-memory buffer)

Investigation:

    $ renacer -- ./data-processor input.csv 2>&1 | grep read | head -3
    read(3, "1,2,3,4,5\n", 10) = 10      # Reading 10 bytes at a time!
    read(3, "6,7,8,9,10\n", 10) = 11
    read(3, "11,12,13\n", 10) = 9

Problem: Buffer size too small (10 bytes per read). For 1MB file, that's 100K syscalls.

Solution: Increase buffer to 4096 bytes, reducing syscalls 400x.

## Anomaly Detection (Sprint 20)

Renacer detects unusual patterns automatically:

    $ renacer -c -- ./app
    System Call Summary:
    ====================
    Syscall          Calls    Errors    Total Time    Avg Time    Anomalies
    openat           1247     247       2.345s        1.881ms     ⚠️ High error rate (19.8%)
    fsync            100      0         12.345s       123.45ms    ⚠️ Slow average (>50ms)
    read             5000     0         3.456s        0.691ms

Anomalies flagged:

- High error rate: `openat` fails 19.8% of the time
- Slow average: `fsync` averaging 123ms per call

### Anomaly Thresholds

Anomaly Type | Threshold | Meaning
--- | --- | ---
High error rate | >5% | More than 5% of calls fail
Slow average | >50ms | Average call duration exceeds 50ms
High variance | p99 > 10x p50 | Extreme outliers present

## Sorting and Analyzing

### Sort by Total Time (Default)

    $ renacer -c -- ./app
    # Sorted by Total Time (highest first)

Shows which syscalls consume most time.

### Sort by Calls

To find most frequently called syscalls, sort output:

    $ renacer -c -- ./app | sort -k2 -n -r
    # Sort by column 2 (Calls), numeric, reversed

### Sort by Errors

To find syscalls with most errors:

    $ renacer -c -- ./app | grep -v "0     " | sort -k3 -n -r
    # Filter out zero errors, sort by column 3 (Errors)

## Combining Statistics with Other Features

### Statistics + Source Correlation

    $ renacer -c --source -- ./app
    System Call Summary:
    ====================
    Syscall          Calls    Total Time    Top Function
    read             5000     3.456s        process_input (src/main.rs:42)
    write            3000     2.345s        flush_output (src/io.rs:89)

Shows which functions are responsible for syscall time.

### Statistics + Function Profiling

    $ renacer -c --function-time -- cargo test
    Function Profiling Summary:
    ========================
    Top 10 Hot Paths (by total time):
      1. cargo::compile  - 45.2% (1.2s, 67 syscalls)
         └─ openat: 34 calls, 890ms
         └─ read: 23 calls, 234ms
         └─ write: 10 calls, 76ms

Breaks down syscall time by function.

### Statistics + Output Formats

Export statistics to JSON for analysis:

    $ renacer -c --format json -- ./app > stats.json
    $ jq '.summary[] | select(.errors > 0)' stats.json
    # Filter to syscalls with errors using jq

## Best Practices

### 1. Use Statistics for Performance Analysis

    # Quick performance overview
    renacer -c -- ./app

Why: Faster than analyzing thousands of individual syscalls.

### 2. Combine with Filtering

    # Focus on file I/O performance
    renacer -c -e 'trace=file' -- ./app

Why: Reduces noise from irrelevant syscalls.

### 3. Check Percentiles for Latency

    # Understand latency distribution
    renacer -c -- ./network-app
    # Look at p50, p90, p99 for outliers

Why: Average can hide important outliers.

### 4. Monitor Error Rates

    # Look for syscalls with errors > 0
    renacer -c -- ./app | grep -v "0     "

Why: Errors indicate bugs or resource issues.

### 5. Export for Long-Term Analysis

    # Export statistics to JSON
    renacer -c --format json -- ./app > stats-$(date +%Y%m%d).json

Why: Track performance regressions over time.

## Common Patterns

### High Call Count, Low Total Time

    Syscall          Calls    Total Time    Avg Time
    getpid           10000    0.123s        0.012ms

Meaning: Called frequently but very fast. Not a bottleneck.

### Low Call Count, High Total Time

    Syscall          Calls    Total Time    Avg Time
    fsync            10       5.678s        567.8ms

Meaning: Infrequent but slow. Major bottleneck.

### High Error Rate

    Syscall          Calls    Errors    Total Time
    openat           1000     500       2.345s      # 50% failure rate!

Meaning: Half of all file opens fail. Check permissions/paths.

### Large Variance (p99 >> p50)

    Syscall          p50      p90      p99      Max
    read             1.2ms    3.4ms    45.6ms   234.5ms

Meaning: Occasional extremely slow reads. Possible disk I/O contention.

## Summary

Statistics mode (`-c`) provides aggregate syscall analysis:

- What: Counts and times syscalls, displays summary table
- Why: Understand overall behavior without trace noise
- How: Add `-c` flag to any renacer command

Key Features:

- Call counts and error counts
- Total time and average time per syscall
- Min/Max timing
- Percentiles (p50/p90/p99) for latency analysis
- SIMD-accelerated computation
- Anomaly detection (high error rates, slow averages)

Best For:

- Performance profiling
- Error analysis
- Identifying bottlenecks
- Comparing before/after optimizations

Combine With:

- Filtering (`-e 'trace=file'`) - Focus on specific syscalls
- Source correlation (`--source`) - See which functions are slow
- Function profiling (`--function-time`) - Per-function breakdown
- Output formats (`--format json`) - Export for analysis

Next Steps:

- Output Formats - Export to JSON/CSV/HTML
- Filtering - Filter syscalls by type or pattern
- Core Concepts Overview - Return to table of contents