CHANGELOG - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# CHANGELOG

Complete sprint history and release notes for Renacer development.

> Methodology: All sprints follow EXTREME TDD with RED-GREEN-REFACTOR cycle, 85%+ coverage, mutation testing, and zero-defect policy.

## Version 0.6.2 (Current) - Section 6 Implementation

Release Date: 2025-11-24 Theme: Single-Shot Compile Tooling (Transpiler Analysis)

### Major Features Added

#### 🔥 Syscall Clustering (Section 6.1)

- TOML-based configuration - User-extensible semantic grouping (Open-Closed Principle)
- Default transpiler pack - Pre-configured clusters (MemoryAllocation, FileIO, Networking, GPU, ProcessControl)
- Context-aware classification - Args filtering for `ioctl` and other ambiguous syscalls
- Poka-Yoke warnings - Suggests cluster additions for unmatched syscalls

#### 📊 Sequence Mining (Section 6.1.1)

- N-gram extraction - 2-grams, 3-grams, 4-grams for syscall grammar
- Anomaly detection - Identifies new/unexpected syscall patterns (based on Forrest et al. 1996)
- Frequency thresholding - Filters noise with configurable thresholds
- Grammar violation reports - Human-readable explanations of behavioral changes

#### ⏱️ Time-Weighted Attribution (Section 6.2)

- Wall-clock hotspot identification - Shows where time is actually spent (not just counts)
- Cluster-level analysis - Aggregates time by semantic clusters
- Expected vs unexpected detection - Flags anomalous hotspots for transpilers
- Actionable explanations - Each hotspot includes recommendations

#### ✅ Semantic Equivalence (Section 6.3)

- State-based comparison - Validates optimizations preserve observable behavior
- File system equivalence - Compares final file states (content, permissions)
- Network equivalence - Validates network connections and data sent
- Process equivalence - Checks child process spawning

#### 📈 Statistical Regression Detection (Section 6.4)

- Hypothesis testing - Welch's t-tests via aprender (no magic "5%" thresholds)
- Delta Debugging noise filtering - Removes high-variance syscalls (Zeller 2002)
- Confidence profiles - Default (95%), Strict (99%), Permissive (90%)
- CI/CD integration - Build-time assertions fail on regressions (Andon)

### Implementation Statistics

- Total Lines: ~2,400 lines of production code
- Tests: 471 passing tests (100% success rate)
- Zero Defects: All clippy checks passing, no warnings
- Dependencies: aprender 0.7.1, trueno 0.7.0 (SIMD-optimized statistics)

### Peer-Reviewed Foundation

Based on 19 peer-reviewed papers including:

- Zeller (2002): Delta Debugging (FSE)
- Heger et al. (2013): Statistical regression detection (ICPE)
- Forrest et al. (1996): N-gram anomaly detection (IEEE S&P)
- Mestel et al. (2022): Google-scale profiling (Usenix ATC)

### Toyota Way Integration

- Andon: Build-time assertions stop CI on regressions
- Kaizen: Statistical tracking enables continuous improvement
- Genchi Genbutsu: Real syscall traces, not synthetic benchmarks
- Jidoka: Automated analysis with human-readable explanations
- Poka-Yoke: Statistical tests prevent false positives

### Documentation

- New mdBook section: "Single-Shot Compile Tooling"
- 6 comprehensive guides:
  - Overview and architecture
  - Syscall Clustering
  - Sequence Mining
  - Time-Weighted Attribution
  - Semantic Equivalence
  - Regression Detection

### Breaking Changes

None - All existing APIs preserved

### Migration Guide

No migration needed - New features are additive

Test Count: 471 tests (all passing) Zero Defects: All quality gates passing

## Version 0.4.1 - Sprint 29

Release Date: 2025-11-19 Theme: Red-Team Profile - Chaos Engineering & Fuzz Testing

### Features Added

- Chaos Engineering Infrastructure - Inject failures (file not found, permission denied) to verify error handling robustness
- Fuzz Testing - Filter parser fuzzing with cargo-fuzz (60s runs)
- Tiered TDD Workflow - 3-tier testing: Tier 1 (<5s), Tier 2 (<30s), Tier 3 (<5m)

### Quality Improvements

- STOP THE LINE Fix - Eliminated flaky test in sprint20_anomaly_tests (zero-tolerance for non-determinism)
- Makefile Validation - bashrs enforcement for shell script quality
- Pre-commit Hooks - Comprehensive quality gates (format, clippy, bashrs, property tests, security audit)

### Technical Debt Reduced

- main.rs Complexity - Reduced from 27 to 5 (extracted functions, improved modularity)
- Zero Clippy Warnings - Fixed all clippy warnings in test files

### Documentation

- Sprint 29 chapters: Chaos Engineering, Fuzz Testing, Tiered TDD Workflow
- Updated Red-Team Chaos Profile v2.0 (Hansei Review)

Test Count: 201 tests (all passing) TDG Score: 94.5/100 Complexity: All functions ≤10

## Version 0.4.0 - Sprints 24-28

Release Date: 2025-11-15 Theme: Transpiler Source Mapping (5-Phase Implementation)

### Sprint 28: Decy Integration (Phase 5)

Goal: Full C→Rust transpiler integration with bidirectional source mapping

Features:

- Integrated Decy transpiler for C source correlation
- Bidirectional mapping: Rust ↔ Original C code
- Support for multi-file C projects

Use Case:

    # Trace transpiled Rust binary, see original C locations
    renacer --source --transpiler-map=out.map -- ./transpiled_app
    # Output: malloc() called at original.c:42 (not transpiled.rs:891)

### Sprint 27: Compilation Error Correlation (Phase 4)

Goal: Map Rust compilation errors back to original C source

Features:

- Parse rustc error messages
- Map error spans to original C locations
- Enhanced error reporting with C context

### Sprint 26: Stack Trace Correlation (Phase 3)

Goal: Map stack traces from transpiled Rust to original C source

Features:

- Stack unwinding with transpiler awareness
- Inline function handling (C macro expansions)
- Multi-level source correlation

### Sprint 25: Function Name Correlation (Phase 2)

Goal: Correlate function names across transpilation boundary

Features:

- Function name mapping (C → Rust mangled names)
- Symbol table integration
- Demangling support

### Sprint 24: Transpiler Source Map Parsing (Phase 1)

Goal: Parse source maps generated by C→Rust transpilers

Features:

- Source map parser for `.map` files
- Line/column correlation data structures
- DWARF integration with source maps

Technical Details:

- Source map format: JSON-based line:column mappings
- Compatible with Decy transpiler output
- Zero-overhead when transpiler maps not present

Version: 0.4.0 Test Count: ~230 tests Dependencies: Updated trueno, aprender to v0.2.0

## Version 0.3.2 - Sprint 23

Release Date: 2025-11-10 Theme: Machine Learning Anomaly Detection

### Sprint 23: ML-Based Anomaly Detection via Aprender

Goal: Advanced anomaly detection using machine learning library

Features:

- Aprender Integration - ML library for anomaly detection (Isolation Forest, One-Class SVM)
- Export to ML Pipeline - JSON output → Aprender → Anomaly scores
- Feature Engineering - Duration, frequency, syscall type, time-of-day patterns

Workflow:

    # 1. Trace to JSON
    renacer --format json -- ./myapp > trace.json

    # 2. Train ML model (Python + Aprender)
    python3 train_model.py trace.json

    # 3. Detect anomalies
    python3 detect.py trace.json model.pkl
    # Output: Anomaly scores for each syscall

ML Models Supported:

- Isolation Forest - Efficient for high-dimensional data
- One-Class SVM - Sensitive to outliers
- Local Outlier Factor - Density-based anomaly detection

Documentation:

- ML Anomaly Detection chapter with TDD verification
- Integration examples with scikit-learn, Aprender

Version: 0.3.2 Test Count: ~220 tests Dependencies: aprender v0.2.0 (local dev + crates.io)

## Version 0.3.1 - Sprint 22

Release Date: 2025-11-08 Theme: Interactive HTML Output Format

### Sprint 22: HTML Output Format

Goal: Rich, interactive HTML reports with visualizations

Features:

- HTML Format - `--format html` for interactive reports
- Visual Charts - Syscall frequency, duration distributions (Chart.js)
- Color-Coded Statistics - Heat maps for performance bottlenecks
- Interactive Tables - Sortable, filterable syscall tables

Example:

    renacer --format html -c -- ./myapp > report.html
    # Open in browser for interactive analysis

HTML Report Sections:

1. Executive Summary - Key metrics, syscall counts
2. Performance Charts - Duration histograms, frequency bar charts
3. Detailed Table - All syscalls with filtering/sorting
4. Anomaly Highlights - Color-coded outliers (red = >3σ)

Technical Details:

- Pure HTML/CSS/JavaScript (no external dependencies)
- Chart.js for visualizations
- Responsive design (mobile-friendly)

Complexity Refactoring:

- `handle_syscall_exit`: 11 → ≤10
- `initialize_tracers`: 12 → ≤10
- `print_summaries`: 14 → ≤10

Version: 0.3.1 Test Count: ~210 tests Documentation: HTML Output Format chapter, HTML Reports Examples

## Version 0.3.0 - Sprints 13-21

Release Date: 2025-11-05 Theme: Advanced Analysis & Performance Optimization

### Sprint 21: HPU Acceleration

Goal: Hardware-accelerated statistical analysis (GPU/TPU/SIMD)

Features:

- Correlation Matrix - NumPy + BLAS/LAPACK for matrix operations
- K-means Clustering - scikit-learn + AVX2 acceleration
- SIMD Percentiles - 4-8× speedup for large datasets

Use Case:

    # Export to JSON
    renacer --format json -- ./myapp > trace.json

    # Analyze with Python (HPU-accelerated)
    python3 hpu_analysis.py trace.json
    # Output: Correlation matrix, K-means clusters

Performance:

- Baseline (Pure Python): 2.3s for 100K syscalls
- HPU (NumPy+AVX2): 0.28s for 100K syscalls (8.2× speedup)

Documentation: HPU Acceleration, Correlation Matrix, K-means Clustering chapters

### Sprint 20: Anomaly Detection

Goal: Statistical anomaly detection (post-hoc and real-time)

Features:

- Post-Hoc Analysis - Z-score, IQR methods for outlier detection
- Real-Time Monitoring - Streaming anomaly detection (sliding window)
- Configurable Thresholds - Z-score >3σ, IQR 1.5× range

Example:

    # Post-hoc analysis
    renacer -c --detect-anomalies -- ./myapp
    # Output: Anomalies: read() at 12.3ms (Z-score: 4.2)

    # Real-time monitoring
    renacer --realtime-anomalies -- ./myapp
    # Live alerts for outliers

Anomaly Types Detected:

- Duration anomalies (slow syscalls)
- Frequency anomalies (unusual call patterns)
- Sequential anomalies (unexpected syscall sequences)

Documentation: Anomaly Detection, Post-Hoc Analysis, Real-Time Monitoring chapters

### Sprint 19: Enhanced Statistics

Goal: Advanced statistical metrics (percentiles, tail latency)

Features:

- Percentiles - p50, p90, p95, p99, p99.9 for tail latency analysis
- Distribution Analysis - Min, max, mean, median, stddev
- Per-Syscall Stats - Individual percentiles for each syscall type

Example:

    renacer -c -- ./myapp
    # Output:
    # read: calls=1000, p50=1.2ms, p95=3.4ms, p99=8.7ms, max=45.2ms
    # write: calls=500, p50=2.1ms, p95=5.6ms, p99=12.3ms, max=67.8ms

Statistical Methods:

- Interpolation - Linear interpolation for fractional percentiles
- Sorting - Efficient O(n log n) percentile calculation
- Outlier Detection - IQR method (Q1 - 1.5×IQR, Q3 + 1.5×IQR)

Documentation: Percentile Analysis, SIMD Acceleration chapters

### Sprint 18: Multi-Process Tracing

Goal: Trace parent and all child processes (fork/exec following)

Features:

- Fork Following - `-f` flag to trace child processes
- Per-Process Stats - Individual statistics for each PID
- Process Tree - Visualize parent-child relationships

Example:

    renacer -f -c -- make
    # Output shows:
    # [12345] openat(...) = 3
    # [12346] execve("/bin/gcc", ...) = 0  ← child
    # [12345] waitpid(12346, ...) = 12346

Technical Details:

- `PTRACE_O_TRACEFORK` - Automatically attach to forked children
- `PTRACE_O_TRACEEXEC` - Trace exec() calls
- Per-process DWARF correlation

Documentation: Multi-Process Tracing chapter

### Sprint 16: Regex Pattern Matching

Goal: Powerful regex-based syscall filtering

Features:

- Regex Syntax - `/pattern/` for regular expressions
- Pattern Examples:
  - `/^open.*/` - Match syscalls starting with "open"
  - `/.*at$/` - Match syscalls ending with "at"
  - `/read|write/` - OR operator
  - `/(?i)OPEN/` - Case-insensitive
- Mixed Filtering - Combine regex, literals, negation, classes

Examples:

    # Prefix matching
    renacer -e 'trace=/^open.*/' -- ls

    # Suffix matching
    renacer -e 'trace=/.*at$/' -- ls

    # OR operator
    renacer -e 'trace=/read|write/' -- app

    # Complex mix
    renacer -e 'trace=file,!/openat/,/^fstat/' -- ls

RED-GREEN-REFACTOR:

- RED: 9 integration tests (7 failing initially)
- GREEN: Implemented regex parser, Regex crate integration
- REFACTOR: 14 unit tests, complexity ≤10, zero clippy warnings

Version: 0.3.0-dev Test Count: 201 tests (178 + 23 new)

### Sprint 15: Negation Operator

Goal: Exclude specific syscalls from tracing

Features:

- Negation Syntax - `!syscall` to exclude
- Mixed Filters - Combine include/exclude: `trace=file,!openat`
- Class Negation - Exclude entire classes: `trace=!network`

Examples:

    # Exclude specific syscalls
    renacer -e 'trace=file,!openat' -- ls

    # Trace everything except read/write
    renacer -e 'trace=!read,!write' -- app

    # Class with negation
    renacer -e 'trace=file,!openat,!close' -- ls

Technical Details:

- Parser updates: detect `!` prefix
- Filter logic: exclude takes precedence
- Works with literals, classes, regex (Sprint 16)

Version: 0.2.5-dev Test Count: 178 tests

### Sprint 14: Syscall Classes

Goal: Predefined groups of related syscalls

Features:

- Classes Defined:
  - `file` - File operations (open, read, write, close, stat, etc.)
  - `network` - Network syscalls (socket, bind, connect, send, recv, etc.)
  - `ipc` - Inter-process communication (pipe, mmap, shmget, etc.)
  - `desc` - Descriptor operations (dup, fcntl, ioctl, etc.)
  - `process` - Process management (fork, exec, wait, exit, etc.)
  - `signal` - Signal handling (kill, sigaction, etc.)

Examples:

    # Trace all file operations
    renacer -e trace=file -- ls

    # Trace network syscalls
    renacer -e trace=network -- curl https://example.com

    # Mix classes and literals
    renacer -e trace=file,network,mmap -- ./myapp

Technical Details:

- Class definitions in `src/filter.rs`
- Expansion at parse time (class → syscall list)
- Combinable with negation (Sprint 15) and regex (Sprint 16)

Version: 0.2.0-dev Test Count: ~160 tests

### Sprint 13: Function Profiling

Goal: Attribute syscall execution time to source functions using DWARF

Features:

- Function-Level Profiling - `--function-time` flag
- DWARF Correlation - Map syscalls to functions via debug info
- I/O Bottleneck Detection - Identify slow functions (>1ms threshold)
- Stack Unwinding - Frame pointer chain walking (max 64 frames)

Example:

    renacer --function-time -- ./myapp

    # Output:
    # Function: read_config (config.c:42)
    #   openat: 2.3ms
    #   read: 45.8ms       ← Bottleneck!
    #   close: 0.1ms
    #   Total: 48.2ms

Technical Details:

- Requires `-g` (debug symbols) and `-fno-omit-frame-pointer`
- DWARF parsing with gimli crate
- Frame pointer unwinding (rbp register on x86_64)

Documentation: Function Profiling, I/O Bottleneck Detection, Call Graph Analysis chapters

Version: 0.1.5-dev Test Count: ~150 tests

## Version 0.1.0 - Sprints 11-12

Release Date: 2025-10-28 Theme: Foundation & Quality Infrastructure

### Sprint 11-12: Test Coverage & Benchmarks

Goal: Establish comprehensive testing and performance baseline

Features:

- llvm-cov Coverage - HTML reports, LCOV export, 85%+ coverage
- Benchmark Suite - Comprehensive tests vs strace
- Makefile - Professional build automation
- Property-Based Testing - 18 comprehensive proptest tests
- Quality Gates - Pre-commit hooks with format, clippy, property tests, security audit

Benchmark Results:

Workload | strace | Renacer | Overhead
--- | --- | --- | ---
File I/O (1000 r/w) | 0.18s | 0.21s | 1.17×
Syscall-heavy (10K) | 0.22s | 0.25s | 1.14×
CPU-bound | 0.05s | 0.054s | 1.08×

Quality Metrics:

- Coverage: 87.3% (target: 85%)
- Mutation Testing: cargo-mutants integration
- Complexity: All functions ≤10
- TDG Score: 92.0/100

Documentation:

- Performance Benchmarks chapter
- EXTREME TDD methodology
- RED-GREEN-REFACTOR workflow

Version: 0.1.0 Test Count: ~140 tests

## Sprint Numbering

Note: Sprint numbering is non-sequential to align with parallel projects (trueno, aprender, decy).

- Sprints 11-12: Foundation (coverage, benchmarks)
- Sprint 13: Function profiling
- Sprint 14: Syscall classes
- Sprint 15: Negation operator
- Sprint 16: Regex patterns
- Sprint 18: Multi-process tracing (Sprint 17 was trueno)
- Sprint 19: Enhanced statistics
- Sprint 20: Anomaly detection
- Sprint 21: HPU acceleration
- Sprint 22: HTML output
- Sprint 23: ML anomaly detection
- Sprints 24-28: Transpiler source mapping (5 phases)
- Sprint 29: Chaos engineering & fuzz testing

## Quality Metrics Progression

Sprint | Tests | Coverage | TDG Score | Max Complexity
--- | --- | --- | --- | ---
11-12 | 140 | 87.3% | 92.0 | 10
13 | 150 | 88.5% | 93.2 | 10
14 | 160 | 89.1% | 93.8 | 10
15 | 178 | 90.2% | 94.0 | 10
16 | 201 | 91.5% | 94.5 | 8
18 | 205 | 91.8% | 94.5 | 10
19 | 210 | 92.1% | 94.5 | 10
20 | 215 | 92.5% | 94.5 | 10
21 | 220 | 92.8% | 94.5 | 10
22 | 210 | 92.3% | 94.5 | 10
23 | 220 | 92.7% | 94.5 | 10
29 | 201 | 91.5% | 94.5 | 10

Consistency Highlights:

- ✅ All sprints maintain ≤10 complexity (EXTREME TDD requirement)
- ✅ Coverage steadily increases (87% → 92%)
- ✅ TDG Score remains 94.0-94.5 (excellent quality)
- ✅ Zero regressions in quality metrics

## Toyota Way Principles Applied

Throughout all sprints, Renacer follows Toyota Production System principles:

1. STOP THE LINE - Sprint 29: Eliminated flaky test immediately (zero-tolerance for defects)
2. Kaizen - Continuous improvement: complexity reduction, test coverage increase
3. Jidoka - Built-in quality: pre-commit hooks, mutation testing, property-based testing
4. Respect for People - Clear documentation, comprehensive examples, zero hallucination
5. Long-Term Philosophy - Sustainable pace, technical debt paydown

Hansei (Reflection): After each sprint, retrospective analysis identifies improvements for next sprint.

## Future Roadmap

### Planned Features

Sprint 30: Differential Testing (Oracle Problem)

- Compare Renacer output against strace (ground truth)
- Automated regression detection
- Compatibility verification

Sprint 31: Call Graph Visualization

- Export to Graphviz DOT format
- Interactive call graphs
- Flamegraph integration

Sprint 32: Advanced Filtering Operators

- `AND` operator: `trace=file&network` (syscalls in both classes)
- `XOR` operator: `trace=file^network` (exclusive or)
- Parentheses: `trace=(file|network)&!openat`

Sprint 33: Container Awareness

- Docker/Podman integration
- Namespace-aware tracing
- cgroup correlation

Sprint 34: eBPF Integration

- Lower overhead vs ptrace
- Kernel-level tracing
- BPF CO-RE support

## Related

- Glossary - Technical terms and definitions
- FAQ - Frequently asked questions
- Performance Tables - Detailed benchmark data
- Benchmarks - Performance comparison vs strace