Golden Trace Validation

The renacer validate subcommand enables regression detection through golden trace comparison. Based on the APR Runtime Model Tracing Support specification (Sprint 50).

Toyota Way: Jidoka

Jidoka means "stop on defect detection." Golden trace validation implements this principle by:

1. Capturing known-good behavior as a baseline (golden trace)
2. Comparing current behavior against the baseline
3. Stopping CI/CD pipelines when regressions are detected

Quick Start

Generate a Baseline

# Create golden trace directory from known-good behavior
renacer validate --generate ./golden -- echo "hello"

This creates:

golden/
  manifest.json     # Trace metadata (version, platform, command)
  syscalls.trace    # Binary syscall sequence (RNTR format)
  timing.stats      # Timing statistics per syscall

Validate Against Baseline

# Compare current behavior against baseline
renacer validate --baseline ./golden -- echo "hello"

Exit code 0: Validation passed Exit code 1: Regression detected

Exit Codes

Configuration Options

Timing Tolerance

By default, timing variations up to 10% are allowed:

# Allow 15% timing variance
renacer validate --baseline ./golden --tolerance 15.0 -- ./myapp

# Strict mode: zero tolerance
renacer validate --baseline ./golden --strict -- ./myapp

Ignore Timing

For behavioral-only validation (ignoring performance):

# Only compare syscall sequences, ignore timing
renacer validate --baseline ./golden --ignore-timing -- ./myapp

Output Formats

# Human-readable text (default)
renacer validate --baseline ./golden --output text -- ./myapp

# JSON for machine parsing
renacer validate --baseline ./golden --output json -- ./myapp

# JUnit XML for CI integration
renacer validate --baseline ./golden --output junit -- ./myapp > results.xml

Output Format Examples

Text Output

=== Validation Report ===

Status: FAILED

Summary:
  Total syscalls compared: 42
  Matches: 40
  Mismatches: 1
  Timing regressions: 1

Syscall Mismatches:
  [DIFFERENT] Index 15: expected 'read', found 'write'

Timing Regressions:
  read: 0.10ms -> 0.12ms (+20.0%)

JSON Output

{
  "status": "failed",
  "total_compared": 42,
  "matches": 40,
  "mismatches": 1,
  "timing_regressions": 1,
  "syscall_details": [
    {
      "index": 15,
      "expected": "read",
      "found": "write",
      "mismatch_type": "different"
    }
  ],
  "timing_details": [
    {
      "syscall": "read",
      "baseline_ms": 0.10,
      "actual_ms": 0.12,
      "delta_percent": 20.0
    }
  ]
}

JUnit XML Output

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="renacer-validate" tests="3" failures="2" errors="0">
  <testcase name="validation-overall" classname="renacer.validate">
    <failure message="Validation failed: 1 mismatches, 1 timing regressions"/>
  </testcase>
  <testcase name="syscall-15" classname="renacer.validate.syscalls">
    <failure message="Expected 'read', found 'write' (different)"/>
  </testcase>
  <testcase name="timing-read" classname="renacer.validate.timing">
    <failure message="Timing regression: 0.10ms -> 0.12ms (+20.0%)"/>
  </testcase>
</testsuite>

Binary Trace Format

The syscalls.trace file uses a binary format for efficient storage:

Header (32 bytes)

Flags Bitfield

Footer

Manifest Format

The manifest.json contains trace metadata:

{
  "version": "1.0.0",
  "renacer_version": "0.5.0",
  "created_at": "1734355200",
  "platform": {
    "os": "linux",
    "arch": "x86_64",
    "kernel": "6.8.0-87-generic"
  },
  "command": ["echo", "hello"],
  "statistics": {
    "total_syscalls": 42,
    "total_duration_ms": 1.234,
    "unique_syscalls": 12
  },
  "tolerance": {
    "timing_percent": 10.0,
    "syscall_sequence": "exact",
    "argument_match": "fuzzy"
  }
}

CI/CD Integration

GitHub Actions

name: Regression Detection

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Renacer
        run: cargo install renacer

      - name: Validate against golden trace
        run: |
          renacer validate --baseline tests/golden/ --output junit -- ./target/release/myapp > results.xml

      - name: Upload test results
        uses: actions/upload-artifact@v4
        with:
          name: validation-results
          path: results.xml

Jenkins Pipeline

pipeline {
    stages {
        stage('Validate') {
            steps {
                sh 'renacer validate --baseline golden/ --output junit -- ./myapp > validation.xml'
            }
            post {
                always {
                    junit 'validation.xml'
                }
            }
        }
    }
}

Comparison Logic

Syscall Sequence Comparison

The validator compares syscall sequences index-by-index:

1. Length check: Different lengths = mismatch
2. Syscall number check: Different syscall at same index = mismatch
3. Order sensitivity: Strict ordering by default

Mismatch types:

• DIFFERENT: Different syscall at same position
• EXTRA: Syscall in actual but not in baseline
• MISSING: Syscall in baseline but not in actual

Timing Regression Detection

For each syscall type, the validator checks:

delta_percent = (actual_mean - baseline_mean) / baseline_mean * 100

if delta_percent > tolerance_percent:
    return TimingRegression

API Usage

use renacer::validate::{
    compare_syscalls, compare_timing, format_json_report,
    ComparisonResult, SyscallTimingStats, TraceSyscallEntry,
    ValidateConfig, ValidateExitCode,
};
use std::collections::HashMap;

// Compare syscall sequences
let baseline = vec![
    TraceSyscallEntry::simple(257, "openat", 50_000),
    TraceSyscallEntry::simple(0, "read", 100_000),
];
let actual = baseline.clone();

let result = compare_syscalls(&baseline, &actual, false)?;
assert!(result.passed);

// Compare timing with tolerance
let mut baseline_stats = HashMap::new();
baseline_stats.insert("read".to_string(), SyscallTimingStats {
    count: 100,
    mean_ns: 100_000,  // 100us baseline
    ..Default::default()
});

let mut actual_stats = HashMap::new();
actual_stats.insert("read".to_string(), SyscallTimingStats {
    count: 100,
    mean_ns: 105_000,  // 5% slower
    ..Default::default()
});

let regressions = compare_timing(&baseline_stats, &actual_stats, 10.0);
assert!(regressions.is_empty());  // Within 10% tolerance

Best Practices

1. Multiple Baseline Runs

Generate baselines from multiple runs to capture natural variance:

for i in {1..5}; do
  renacer validate --generate ./golden-run-$i -- ./myapp
done
# Then merge statistics

2. Platform-Specific Baselines

Kernel and hardware differences affect syscall behavior:

# Store baselines per platform
renacer validate --generate ./golden/linux-x86_64/ -- ./myapp
renacer validate --generate ./golden/linux-arm64/ -- ./myapp

3. Lenient Tolerance for CI

Use higher tolerance in CI to reduce false positives from runner variance:

renacer validate --baseline ./golden --tolerance 25.0 -- ./myapp

4. Strict Mode for Security

For security-critical validation, use strict mode:

renacer validate --baseline ./golden --strict -- ./security-critical-app

Related

• Regression Detection - Statistical hypothesis testing
• Semantic Equivalence - Behavioral equivalence checking
• Syscall Clustering - Group related syscalls

Toyota Way Principles Applied