OpenTelemetry Integration - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# OpenTelemetry Integration

Renacer integrates with OpenTelemetry (OTLP) to export syscall traces as distributed tracing spans. This enables seamless integration with observability backends like Jaeger, Grafana Tempo, Elastic APM, and Honeycomb.

## Overview

OpenTelemetry integration allows you to:

- Export syscall traces as standardized OTLP spans
- View traces in familiar observability tools
- Correlate system calls with application traces
- Build end-to-end observability across your stack

## Quick Start

### 1. Start Jaeger (for local testing)

    docker run -d --name jaeger \
      -p 16686:16686 \
      -p 4317:4317 \
      -p 4318:4318 \
      jaegertracing/all-in-one:latest

### 2. Trace with OTLP Export

    # Export via gRPC (default port 4317)
    renacer --otlp-endpoint http://localhost:4317 -- ls -la

    # Export via HTTP (port 4318)
    renacer --otlp-endpoint http://localhost:4318 --otlp-protocol http -- ls -la

### 3. View in Jaeger

Open http://localhost:16686 and select service "renacer" to view traces.

## OTLP Protocols

Renacer supports both OTLP protocols:

### gRPC (Default)

    renacer --otlp-endpoint http://localhost:4317 -- ./my-app

Advantages:

- Better performance for high-volume traces
- Built-in compression and flow control
- Standard port: 4317

### HTTP/protobuf

    renacer --otlp-endpoint http://localhost:4318 --otlp-protocol http -- ./my-app

Advantages:

- Simpler firewall configuration
- Works with HTTP proxies
- Standard port: 4318

## Span Structure

Renacer creates a hierarchical span structure:

    Root Span (Process)
    ├── Syscall Span: openat
    ├── Syscall Span: read
    ├── Syscall Span: write
    └── Syscall Span: close

### Root Span Attributes

    {
      "service.name": "renacer",
      "process.pid": 12345,
      "process.command": "./my-app --flag",
      "process.executable": "/path/to/my-app"
    }

### Syscall Span Attributes

    {
      "syscall.name": "openat",
      "syscall.number": 257,
      "syscall.args": "AT_FDCWD, \"/etc/passwd\", O_RDONLY",
      "syscall.result": "3",
      "syscall.duration_us": 42,
      "source.file": "src/main.rs",
      "source.line": 15,
      "source.function": "read_config"
    }

## Backend Configuration

### Jaeger

    # Local Jaeger instance
    renacer --otlp-endpoint http://localhost:4317 -- ./app

    # Remote Jaeger
    renacer --otlp-endpoint https://jaeger.example.com:4317 -- ./app

### Grafana Tempo

    # Tempo with gRPC
    renacer --otlp-endpoint http://tempo:4317 -- ./app

    # Tempo with HTTP
    renacer --otlp-endpoint http://tempo:4318 --otlp-protocol http -- ./app

### Elastic APM

    renacer --otlp-endpoint https://apm.elastic.co:443 \
      --otlp-headers "Authorization=Bearer YOUR_TOKEN" \
      -- ./app

### Honeycomb

    renacer --otlp-endpoint https://api.honeycomb.io:443 \
      --otlp-headers "x-honeycomb-team=YOUR_API_KEY,x-honeycomb-dataset=renacer" \
      --otlp-protocol http \
      -- ./app

## Custom Headers

Use `--otlp-headers` for authentication:

    renacer --otlp-endpoint https://api.example.com:4317 \
      --otlp-headers "Authorization=Bearer token123,X-Custom=value" \
      -- ./app

Headers are comma-separated key=value pairs.

## Performance Considerations

### Batching

Renacer batches spans before export to reduce network overhead:

    # Default batch size: 512 spans
    renacer --otlp-endpoint http://localhost:4317 -- ./app

    # Custom batch size (Sprint 36 feature)
    # Controlled via environment variable RENACER_OTLP_BATCH_SIZE
    export RENACER_OTLP_BATCH_SIZE=1024
    renacer --otlp-endpoint http://localhost:4317 -- ./app

Batching reduces network overhead by 40-60%.

### Async Export

OTLP export is asynchronous and doesn't block tracing:

- Spans are queued in memory
- Background thread handles export
- Zero blocking on syscall tracing path
- Automatic retry on transient failures

### Overhead

With Sprint 36 optimizations:

- Basic OTLP export: <5% overhead
- Full observability stack: <10% overhead

See Performance Optimization for details.

## Source Correlation

When tracing programs with debug symbols:

    renacer --source --otlp-endpoint http://localhost:4317 -- ./my-app

Spans include source location attributes:

- `source.file`: Source file path
- `source.line`: Line number
- `source.function`: Function name (when available)

This enables powerful correlation in observability UIs.

## Filtering with OTLP

Combine filtering with OTLP export:

    # Export only file operations
    renacer --syscall-class file --otlp-endpoint http://localhost:4317 -- ./app

    # Export only slow syscalls (>1ms)
    renacer --filter-duration-gt 1000 --otlp-endpoint http://localhost:4317 -- ./app

## Multi-Process Tracing

Trace forked processes with OTLP:

    renacer -f --otlp-endpoint http://localhost:4317 -- ./parent-app

Each process gets its own root span with unique `process.pid`.

## Troubleshooting

### Connection Refused

    Error: Failed to export spans: connection refused

Solution: Verify OTLP endpoint is running and accessible:

    # Test gRPC endpoint
    grpcurl -plaintext localhost:4317 list

    # Test HTTP endpoint
    curl http://localhost:4318/v1/traces

### Authentication Failed

    Error: OTLP export failed: 401 Unauthorized

Solution: Check authentication headers:

    renacer --otlp-endpoint https://api.example.com \
      --otlp-headers "Authorization=Bearer YOUR_VALID_TOKEN" \
      -- ./app

### No Spans in Backend

Checklist:

1. Is the backend receiving data? Check backend logs
2. Is the service name correct? Default is "renacer"
3. Are spans being filtered? Check backend filters
4. Is batching delaying export? Wait a few seconds

### Protocol Mismatch

    Error: Protocol error: expected gRPC, got HTTP

Solution: Match protocol to endpoint:

    # Port 4317 = gRPC (default)
    renacer --otlp-endpoint http://localhost:4317 -- ./app

    # Port 4318 = HTTP
    renacer --otlp-endpoint http://localhost:4318 --otlp-protocol http -- ./app

## Example: Full Observability Stack

Run Renacer with complete observability:

    renacer \
      --source \
      --function-time \
      --stats \
      --anomaly-detection \
      --otlp-endpoint http://localhost:4317 \
      -- cargo test

This exports:

- ✅ All syscalls with source correlation
- ✅ Function-level profiling data
- ✅ Statistical summaries
- ✅ Anomaly alerts
- ✅ OTLP spans to Jaeger/Tempo

## Next Steps

- Distributed Tracing - Link traces across services
- Performance Optimization - Minimize overhead
- Transpiler Integration - Trace transpiled code