Percentile Analysis - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# Percentile Analysis

Percentile analysis provides statistical insights into syscall duration distribution, helping identify outliers and performance variability beyond simple averages.

> TDD-Verified: Percentile calculations tested in `tests/sprint19_enhanced_stats_tests.rs`

> Parent Chapter: See Statistical Analysis for overview

## Overview

Percentiles answer the question: "What percentage of syscalls complete within X microseconds?"

- p50 (median) - 50% of syscalls complete faster than this
- p95 - 95% of syscalls complete faster than this (95th percentile)
- p99 - 99% of syscalls complete faster than this (outlier threshold)
- p99.9 - 99.9% complete faster (rare outliers)

Why percentiles matter:

- Averages hide outliers - p99 reveals tail latency
- SLO/SLA compliance - "99% of requests <100ms"
- Performance regression detection - p99 degradation indicates problems

## Calculating Percentiles

Renacer's statistics mode (`-c`) provides percentile analysis via external post-processing:

    $ renacer -c --format json -- ./myapp > stats.json

Then calculate percentiles with jq:

    #!/bin/bash
    # Extract syscall durations, calculate p50/p95/p99

    jq -r '.syscalls[] | .duration_ns' stats.json | \
      sort -n | \
      awk '
        {durations[NR] = $1}
        END {
          p50 = durations[int(NR * 0.50)]
          p95 = durations[int(NR * 0.95)]
          p99 = durations[int(NR * 0.99)]
          printf "p50: %d ns\np95: %d ns\np99: %d ns\n", p50, p95, p99
        }
      '

Example Output:

    p50: 1234 ns
    p95: 5678 ns
    p99: 12345 ns

Tested by: Sprint 19 enhanced statistics tests

## Practical Examples

### Example 1: Database Query Latency

    $ renacer -c --format json -e trace=network -- ./db-app > db-stats.json

Calculate percentiles:

    p50: 2000 μs (median - typical query)
    p95: 8000 μs (95% complete within 8ms)
    p99: 25000 μs (99% complete within 25ms)

Analysis:

- Median (p50) is fast (2ms)
- p99 is 12.5× slower than median → high variability!
- Action: Investigate p99 outliers (cache misses, slow queries)

### Example 2: Identifying Tail Latency

Before optimization:

    p50: 100 μs
    p95: 500 μs (5× median)
    p99: 5000 μs (50× median!) ← High tail latency

After adding caching:

    p50: 95 μs (5% faster)
    p95: 450 μs (10% faster)
    p99: 800 μs (6.25× faster!) ← Tail latency improved!

Result: p99 improvement shows caching eliminated outliers ✅

## Advanced Workflows

### Percentile Heatmaps

Generate percentile distributions over time:

    #!/usr/bin/env python3
    import json
    import sys
    from collections import defaultdict

    with open(sys.argv[1]) as f:
        data = json.load(f)

    # Group by syscall type
    by_syscall = defaultdict(list)
    for sc in data['syscalls']:
        by_syscall[sc['name']].append(sc['duration_ns'])

    # Calculate percentiles per syscall
    for name, durations in sorted(by_syscall.items()):
        durations.sort()
        n = len(durations)
        p50 = durations[int(n * 0.50)] if n > 0 else 0
        p95 = durations[int(n * 0.95)] if n > 0 else 0
        p99 = durations[int(n * 0.99)] if n > 0 else 0

        print(f"{name:15s} p50:{p50:6d} p95:{p95:6d} p99:{p99:6d} ns")

Output:

    read            p50:  1234 p95:  5678 p99: 12345 ns
    write           p50:   890 p95:  4567 p99:  9876 ns
    openat          p50:  2345 p95:  6789 p99: 23456 ns

### SLO Compliance Checking

Check if p99 latency meets SLO (e.g., <10ms):

    $ jq -r '.syscalls[] | .duration_ns' stats.json | \
      sort -n | awk '
        END {
          p99 = $0  # Last value after sort
          slo_ns = 10000000  # 10ms in nanoseconds
          if (p99 > slo_ns) {
            printf "❌ SLO violation: p99 = %.2f ms (limit: 10 ms)\n", p99/1000000
            exit 1
          } else {
            printf "✅ SLO met: p99 = %.2f ms\n", p99/1000000
          }
        }
      '

## Summary

Percentile analysis provides:

- ✅ Tail latency visibility (p95, p99, p99.9)
- ✅ SLO/SLA compliance validation
- ✅ Outlier detection beyond averages
- ✅ Performance regression early warning

Workflow: Export JSON → Calculate percentiles with jq/awk/Python

All statistics tested in: `tests/sprint19_enhanced_stats_tests.rs`

## Related

- Statistical Analysis - Parent chapter
- SIMD Acceleration - Fast percentile calculations
- Export to JSON/CSV - Data export workflows