Performance Optimization

Renacer is designed for production use with minimal overhead. Sprint 36 introduced comprehensive performance optimizations that reduce overhead to <5% for basic tracing and <10% for the full observability stack.

Performance Goals

Target Overhead

Baseline Comparison

vs. traditional strace:

• strace: 8-12% overhead (basic tracing)
• Renacer: 3-4% overhead (basic tracing) → 2-3x faster

Sprint 36 Optimizations

Sprint 36 delivered four major performance enhancements:

1. Memory Pool (span_pool.rs)

What: Object pooling for OTLP span allocations

Benefit: 20-30% reduction in allocations

How it works:

// Instead of allocating each span individually
let span = Box::new(Span::new(...));  // ❌ Expensive

// Reuse pre-allocated spans from pool
let span = span_pool.acquire();       // ✅ Fast
span.reset_and_configure(...);

Configuration:

# Set pool capacity (default: 1024)
export RENACER_SPAN_POOL_SIZE=2048

renacer --otlp-endpoint http://localhost:4317 -- ./app

Pool Statistics:

# Enable pool statistics (debug builds)
export RENACER_POOL_STATS=1

renacer --otlp-endpoint http://localhost:4317 -- ./app

# Output:
# Span Pool Statistics:
#   Hits: 15234 (98.5%)
#   Misses: 234 (1.5%)
#   Pool Efficiency: Excellent

2. Zero-Copy Strings (Cow<'static, str>)

What: Avoid allocating static strings

Benefit: 10-15% memory reduction

How it works:

// Old: Always allocate
let syscall_name = format!("openat");  // ❌ Heap allocation

// New: Use static string when possible
let syscall_name: Cow<'static, str> = "openat".into();  // ✅ Zero-copy

What's optimized:

• Syscall names (335 syscalls → all static)
• Span attribute keys (syscall.name, source.file, etc.)
• Common attribute values (O_RDONLY, SEEK_SET, etc.)

3. Lazy Span Creation (lazy_span.rs)

What: Defer expensive work until spans are exported

Benefit: 5-10% overhead reduction

How it works:

// Old: Build span immediately
let span = Span::new();
span.set_name("openat");              // ❌ Work done even if not exported
span.set_attribute("syscall.args", ...);

// New: Lazy builder pattern
let span = LazySpan::builder()
    .name("openat")                   // ✅ Deferred
    .attribute("syscall.args", ...)
    .build_if_needed();               // Only built when exporting

When spans are never exported:

• Features disabled: No OTLP flag → span builder is free
• Cancelled spans: Filtered syscalls → no work done

4. Batch OTLP Export

What: Send spans in batches instead of individually

Benefit: 40-60% network overhead reduction

How it works:

// Old: Export each span individually
for span in spans {
    otlp_client.export(span).await;   // ❌ Many network calls
}

// New: Batch export
otlp_client.export_batch(spans).await; // ✅ Single network call

Configuration:

# Set batch size (default: 512)
export RENACER_OTLP_BATCH_SIZE=1024

# Set batch timeout (default: 5s)
export RENACER_OTLP_BATCH_TIMEOUT=10

renacer --otlp-endpoint http://localhost:4317 -- ./app

Trade-offs:

• Larger batches → Lower network overhead, higher latency
• Smaller batches → Higher network overhead, lower latency

Performance Presets

Renacer provides three performance presets:

Balanced (Default)

renacer --otlp-endpoint http://localhost:4317 -- ./app

Settings:

• Span pool: 1024 spans
• Batch size: 512 spans
• Batch timeout: 5s

Best for: Most production workloads

Aggressive (Max Throughput)

export RENACER_PERF_PRESET=aggressive

renacer --otlp-endpoint http://localhost:4317 -- ./app

Settings:

• Span pool: 4096 spans
• Batch size: 2048 spans
• Batch timeout: 10s

Best for:

• High-throughput services (>10K syscalls/sec)
• Batch processing workloads
• Lower priority for real-time visibility

Low-Latency (Min Delay)

export RENACER_PERF_PRESET=low_latency

renacer --otlp-endpoint http://localhost:4317 -- ./app

Settings:

• Span pool: 512 spans
• Batch size: 128 spans
• Batch timeout: 1s

Best for:

• Interactive applications
• Debugging with real-time feedback
• Low syscall rate (<1K syscalls/sec)

Benchmarking

Sprint 36 includes a comprehensive benchmark suite.

Running Benchmarks

# Run all benchmarks
cargo bench

# Run specific benchmark
cargo bench --bench syscall_overhead
cargo bench --bench otlp_export
cargo bench --bench memory_pool

Benchmark Suite

1. Syscall Overhead (benches/syscall_overhead.rs)

Measures tracing overhead:

cargo bench --bench syscall_overhead

Output:

syscall_overhead/baseline          time: 1.2 µs
syscall_overhead/renacer_basic     time: 1.25 µs (+4.2%)
syscall_overhead/renacer_source    time: 1.32 µs (+10%)
syscall_overhead/renacer_full      time: 1.42 µs (+18%)

Scenarios:

• baseline: No tracing (native syscall)
• renacer_basic: Basic tracing
• renacer_source: With DWARF correlation
• renacer_full: Full stack (OTLP + profiling + stats)

2. OTLP Export (benches/otlp_export.rs)

Measures export throughput:

cargo bench --bench otlp_export

Output:

otlp_export/individual            time: 125 µs/span
otlp_export/batched_512           time: 2.1 µs/span (60x faster!)
otlp_export/batched_2048          time: 0.8 µs/span (156x faster!)

3. Memory Pool (benches/memory_pool.rs)

Measures allocation performance:

cargo bench --bench memory_pool

Output:

memory_pool/direct_alloc          time: 85 ns/span
memory_pool/pooled_alloc          time: 12 ns/span (7x faster!)
memory_pool/pool_acquire_hit      time: 8 ns
memory_pool/pool_acquire_miss     time: 90 ns

Interpreting Results

Good Performance:

• Basic overhead: <5%
• Pool hit rate: >95%
• Batch throughput: >100K spans/sec

Needs Tuning:

• Basic overhead: >8%
• Pool hit rate: <80%
• Batch throughput: <50K spans/sec

Profiling Renacer Itself

Profile Renacer's performance:

1. Using perf

# Profile Renacer while tracing
perf record -g -- renacer --otlp-endpoint http://localhost:4317 -- ./app

# View flamegraph
perf script | stackcollapse-perf.pl | flamegraph.pl > renacer-profile.svg

2. Using Renacer's Self-Profiling

# Enable self-profiling
export RENACER_PROFILE_SELF=1

renacer --otlp-endpoint http://localhost:4317 -- ./app

# Output:
# Renacer Self-Profile:
#   Time in ptrace:      45.2% (1.2s)
#   Time in DWARF:       12.3% (330ms)
#   Time in OTLP export: 8.1% (220ms)
#   Time in pool ops:    2.1% (55ms)
#   Other:               32.3% (870ms)

3. Memory Profiling

# Track allocations
export RENACER_TRACK_ALLOCS=1

renacer --otlp-endpoint http://localhost:4317 -- ./app

# Output:
# Memory Profile:
#   Peak memory: 15.2 MB
#   Total allocations: 45,234
#   Pool reuse: 98.5%
#   Zero-copy strings: 87.3%

Optimization Tips

1. Enable Only Needed Features

# ✅ Good: Only what you need
renacer --source -- ./app

# ❌ Bad: Everything enabled
renacer --source --function-time --stats --anomaly-detection --hpu -- ./app

Overhead by feature:

• Basic tracing: +3%
• Source correlation: +2%
• Function profiling: +3%
• Statistics: +1%
• Anomaly detection: +1%
• OTLP export: +2%

2. Use Appropriate Filters

# ✅ Good: Filter at source
renacer --syscall-class file -- ./app

# ❌ Bad: Trace everything, filter later
renacer -- ./app | grep "open"

3. Tune Batch Size

# High syscall rate (>10K/sec)
export RENACER_OTLP_BATCH_SIZE=2048

# Low syscall rate (<1K/sec)
export RENACER_OTLP_BATCH_SIZE=128

renacer --otlp-endpoint http://localhost:4317 -- ./app

4. Increase Pool Size for High Load

# Default: 1024 spans
# For >5K syscalls/sec:
export RENACER_SPAN_POOL_SIZE=4096

renacer --otlp-endpoint http://localhost:4317 -- ./app

5. Use Local OTLP Collector

# ✅ Good: Local collector (low latency)
renacer --otlp-endpoint http://localhost:4317 -- ./app

# ❌ Bad: Remote collector (high latency)
renacer --otlp-endpoint https://remote-collector.example.com:4317 -- ./app

Use OpenTelemetry Collector as local aggregator:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp:
    endpoint: remote-backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

6. Compile with Release Mode

# Always compile traced programs with optimizations
rustc -C opt-level=3 app.rs -o app

# But keep debug symbols for source correlation
rustc -C opt-level=3 -g app.rs -o app

Regression Detection

Prevent performance regressions with automated checks:

CI/CD Integration

# .github/workflows/perf.yml
name: Performance Tests

on: [pull_request]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Run benchmarks
        run: cargo bench --bench syscall_overhead -- --save-baseline main

      - name: Compare with baseline
        run: |
          cargo bench --bench syscall_overhead -- --baseline main
          # Fail if overhead increased >5%

Manual Comparison

# Save baseline
cargo bench -- --save-baseline before_changes

# Make changes...

# Compare
cargo bench -- --baseline before_changes

Troubleshooting Performance

High Overhead (>15%)

Symptoms:

• Application runs much slower under Renacer
• Syscall overhead >15%

Diagnosis:

# Enable self-profiling
export RENACER_PROFILE_SELF=1
renacer -- ./app

Common causes:

1. Too many syscalls: Filter unnecessary ones

   renacer --syscall-class file -- ./app  # Only file I/O
   

2. DWARF parsing slow: Use transpiler maps instead

   renacer --transpiler-map app.map.json -- ./app
   

3. Remote OTLP endpoint: Use local collector

   renacer --otlp-endpoint http://localhost:4317 -- ./app
   

High Memory Usage

Symptoms:

• Renacer uses >100MB memory
• OOM errors on long-running traces

Diagnosis:

export RENACER_TRACK_ALLOCS=1
renacer -- ./app

Solutions:

1. Reduce span pool size:

   export RENACER_SPAN_POOL_SIZE=512
   

2. Increase batch frequency:

   export RENACER_OTLP_BATCH_TIMEOUT=1  # Flush every 1s
   

3. Filter syscalls:

   renacer --syscall-class file -- ./app
   

Poor Pool Hit Rate (<80%)

Symptoms:

Span Pool Statistics:
  Hits: 12000 (75%)
  Misses: 4000 (25%)

Solution: Increase pool size

export RENACER_SPAN_POOL_SIZE=2048

OTLP Export Bottleneck

Symptoms:

• Tracing fast, but export slow
• Spans buffered in memory

Diagnosis:

export RENACER_PROFILE_SELF=1
renacer --otlp-endpoint http://localhost:4317 -- ./app
# Look for high "Time in OTLP export"

Solutions:

1. Increase batch size:

   export RENACER_OTLP_BATCH_SIZE=2048
   

2. Use gRPC instead of HTTP:

   renacer --otlp-endpoint http://localhost:4317 -- ./app  # gRPC (faster)
   

3. Use local collector:

   # Run otel-collector locally
   docker run -p 4317:4317 otel/opentelemetry-collector
   

Performance Best Practices Summary

Do's ✅

1. Filter aggressively - Only trace what you need
2. Use local OTLP collector - Minimize network latency
3. Tune batch sizes - Match your syscall rate
4. Enable only needed features - Each adds overhead
5. Compile with optimizations - Use -C opt-level=3
6. Monitor pool hit rate - Adjust size as needed
7. Run benchmarks regularly - Catch regressions early

Don'ts ❌

1. Don't enable all features - Unless debugging
2. Don't use remote OTLP endpoints - Use local collector
3. Don't trace without filtering - Filter syscalls
4. Don't use tiny batch sizes - Increases network overhead
5. Don't ignore pool statistics - They guide tuning
6. Don't run in debug mode - Always use release builds
7. Don't skip benchmarking - Measure, don't guess

Performance Comparison Table

Next Steps

• Reference: Benchmarks - Detailed benchmark results
• OpenTelemetry Integration - OTLP export configuration
• Distributed Tracing - Low-overhead distributed tracing