CHANGELOG

Complete sprint history and release notes for Renacer development.

Methodology: All sprints follow EXTREME TDD with RED-GREEN-REFACTOR cycle, 85%+ coverage, mutation testing, and zero-defect policy.

Version 0.6.2 (Current) - Section 6 Implementation

Release Date: 2025-11-24 Theme: Single-Shot Compile Tooling (Transpiler Analysis)

Major Features Added

🔥 Syscall Clustering (Section 6.1)

TOML-based configuration - User-extensible semantic grouping (Open-Closed Principle)
Default transpiler pack - Pre-configured clusters (MemoryAllocation, FileIO, Networking, GPU, ProcessControl)
Context-aware classification - Args filtering for ioctl and other ambiguous syscalls
Poka-Yoke warnings - Suggests cluster additions for unmatched syscalls

📊 Sequence Mining (Section 6.1.1)

N-gram extraction - 2-grams, 3-grams, 4-grams for syscall grammar
Anomaly detection - Identifies new/unexpected syscall patterns (based on Forrest et al. 1996)
Frequency thresholding - Filters noise with configurable thresholds
Grammar violation reports - Human-readable explanations of behavioral changes

⏱️ Time-Weighted Attribution (Section 6.2)

Wall-clock hotspot identification - Shows where time is actually spent (not just counts)
Cluster-level analysis - Aggregates time by semantic clusters
Expected vs unexpected detection - Flags anomalous hotspots for transpilers
Actionable explanations - Each hotspot includes recommendations

✅ Semantic Equivalence (Section 6.3)

State-based comparison - Validates optimizations preserve observable behavior
File system equivalence - Compares final file states (content, permissions)
Network equivalence - Validates network connections and data sent
Process equivalence - Checks child process spawning

📈 Statistical Regression Detection (Section 6.4)

Hypothesis testing - Welch's t-tests via aprender (no magic "5%" thresholds)
Delta Debugging noise filtering - Removes high-variance syscalls (Zeller 2002)
Confidence profiles - Default (95%), Strict (99%), Permissive (90%)
CI/CD integration - Build-time assertions fail on regressions (Andon)

Implementation Statistics

Total Lines: ~2,400 lines of production code
Tests: 471 passing tests (100% success rate)
Zero Defects: All clippy checks passing, no warnings
Dependencies: aprender 0.7.1, trueno 0.7.0 (SIMD-optimized statistics)

Peer-Reviewed Foundation

Based on 19 peer-reviewed papers including:

Zeller (2002): Delta Debugging (FSE)
Heger et al. (2013): Statistical regression detection (ICPE)
Forrest et al. (1996): N-gram anomaly detection (IEEE S&P)
Mestel et al. (2022): Google-scale profiling (Usenix ATC)

Toyota Way Integration

Andon: Build-time assertions stop CI on regressions
Kaizen: Statistical tracking enables continuous improvement
Genchi Genbutsu: Real syscall traces, not synthetic benchmarks
Jidoka: Automated analysis with human-readable explanations
Poka-Yoke: Statistical tests prevent false positives

Documentation

New mdBook section: "Single-Shot Compile Tooling"
6 comprehensive guides:
- Overview and architecture
- Syscall Clustering
- Sequence Mining
- Time-Weighted Attribution
- Semantic Equivalence
- Regression Detection

Breaking Changes

None - All existing APIs preserved

Migration Guide

No migration needed - New features are additive

Test Count: 471 tests (all passing) Zero Defects: All quality gates passing

Version 0.4.1 - Sprint 29

Release Date: 2025-11-19 Theme: Red-Team Profile - Chaos Engineering & Fuzz Testing

Features Added

Chaos Engineering Infrastructure - Inject failures (file not found, permission denied) to verify error handling robustness
Fuzz Testing - Filter parser fuzzing with cargo-fuzz (60s runs)
Tiered TDD Workflow - 3-tier testing: Tier 1 (<5s), Tier 2 (<30s), Tier 3 (<5m)

Quality Improvements

STOP THE LINE Fix - Eliminated flaky test in sprint20_anomaly_tests (zero-tolerance for non-determinism)
Makefile Validation - bashrs enforcement for shell script quality
Pre-commit Hooks - Comprehensive quality gates (format, clippy, bashrs, property tests, security audit)

Technical Debt Reduced

main.rs Complexity - Reduced from 27 to 5 (extracted functions, improved modularity)
Zero Clippy Warnings - Fixed all clippy warnings in test files

Documentation

Sprint 29 chapters: Chaos Engineering, Fuzz Testing, Tiered TDD Workflow
Updated Red-Team Chaos Profile v2.0 (Hansei Review)

Test Count: 201 tests (all passing) TDG Score: 94.5/100 Complexity: All functions ≤10

Version 0.4.0 - Sprints 24-28

Release Date: 2025-11-15 Theme: Transpiler Source Mapping (5-Phase Implementation)

Sprint 28: Decy Integration (Phase 5)

Goal: Full C→Rust transpiler integration with bidirectional source mapping

Features:

Integrated Decy transpiler for C source correlation
Bidirectional mapping: Rust ↔ Original C code
Support for multi-file C projects

Use Case:

# Trace transpiled Rust binary, see original C locations
renacer --source --transpiler-map=out.map -- ./transpiled_app
# Output: malloc() called at original.c:42 (not transpiled.rs:891)

Sprint 27: Compilation Error Correlation (Phase 4)

Goal: Map Rust compilation errors back to original C source

Features:

Parse rustc error messages
Map error spans to original C locations
Enhanced error reporting with C context

Sprint 26: Stack Trace Correlation (Phase 3)

Goal: Map stack traces from transpiled Rust to original C source

Features:

Stack unwinding with transpiler awareness
Inline function handling (C macro expansions)
Multi-level source correlation

Sprint 25: Function Name Correlation (Phase 2)

Goal: Correlate function names across transpilation boundary

Features:

Function name mapping (C → Rust mangled names)
Symbol table integration
Demangling support

Sprint 24: Transpiler Source Map Parsing (Phase 1)

Goal: Parse source maps generated by C→Rust transpilers

Features:

Source map parser for .map files
Line/column correlation data structures
DWARF integration with source maps

Technical Details:

Source map format: JSON-based line:column mappings
Compatible with Decy transpiler output
Zero-overhead when transpiler maps not present

Version: 0.4.0 Test Count: ~230 tests Dependencies: Updated trueno, aprender to v0.2.0

Version 0.3.2 - Sprint 23

Release Date: 2025-11-10 Theme: Machine Learning Anomaly Detection

Sprint 23: ML-Based Anomaly Detection via Aprender

Goal: Advanced anomaly detection using machine learning library

Features:

Aprender Integration - ML library for anomaly detection (Isolation Forest, One-Class SVM)
Export to ML Pipeline - JSON output → Aprender → Anomaly scores
Feature Engineering - Duration, frequency, syscall type, time-of-day patterns

Workflow:

# 1. Trace to JSON
renacer --format json -- ./myapp > trace.json

# 2. Train ML model (Python + Aprender)
python3 train_model.py trace.json

# 3. Detect anomalies
python3 detect.py trace.json model.pkl
# Output: Anomaly scores for each syscall

ML Models Supported:

Isolation Forest - Efficient for high-dimensional data
One-Class SVM - Sensitive to outliers
Local Outlier Factor - Density-based anomaly detection

Documentation:

ML Anomaly Detection chapter with TDD verification
Integration examples with scikit-learn, Aprender

Version: 0.3.2 Test Count: ~220 tests Dependencies: aprender v0.2.0 (local dev + crates.io)

Version 0.3.1 - Sprint 22

Release Date: 2025-11-08 Theme: Interactive HTML Output Format

Sprint 22: HTML Output Format

Goal: Rich, interactive HTML reports with visualizations

Features:

HTML Format - --format html for interactive reports
Visual Charts - Syscall frequency, duration distributions (Chart.js)
Color-Coded Statistics - Heat maps for performance bottlenecks
Interactive Tables - Sortable, filterable syscall tables

Example:

renacer --format html -c -- ./myapp > report.html
# Open in browser for interactive analysis

HTML Report Sections:

Executive Summary - Key metrics, syscall counts
Performance Charts - Duration histograms, frequency bar charts
Detailed Table - All syscalls with filtering/sorting
Anomaly Highlights - Color-coded outliers (red = >3σ)

Technical Details:

Pure HTML/CSS/JavaScript (no external dependencies)
Chart.js for visualizations
Responsive design (mobile-friendly)

Complexity Refactoring:

handle_syscall_exit: 11 → ≤10
initialize_tracers: 12 → ≤10
print_summaries: 14 → ≤10

Version: 0.3.1 Test Count: ~210 tests Documentation: HTML Output Format chapter, HTML Reports Examples

Version 0.3.0 - Sprints 13-21

Release Date: 2025-11-05 Theme: Advanced Analysis & Performance Optimization

Sprint 21: HPU Acceleration

Goal: Hardware-accelerated statistical analysis (GPU/TPU/SIMD)

Features:

Correlation Matrix - NumPy + BLAS/LAPACK for matrix operations
K-means Clustering - scikit-learn + AVX2 acceleration
SIMD Percentiles - 4-8× speedup for large datasets

Use Case:

# Export to JSON
renacer --format json -- ./myapp > trace.json

# Analyze with Python (HPU-accelerated)
python3 hpu_analysis.py trace.json
# Output: Correlation matrix, K-means clusters

Performance:

Baseline (Pure Python): 2.3s for 100K syscalls
HPU (NumPy+AVX2): 0.28s for 100K syscalls (8.2× speedup)

Documentation: HPU Acceleration, Correlation Matrix, K-means Clustering chapters

Sprint 20: Anomaly Detection

Goal: Statistical anomaly detection (post-hoc and real-time)

Features:

Post-Hoc Analysis - Z-score, IQR methods for outlier detection
Real-Time Monitoring - Streaming anomaly detection (sliding window)
Configurable Thresholds - Z-score >3σ, IQR 1.5× range

Example:

# Post-hoc analysis
renacer -c --detect-anomalies -- ./myapp
# Output: Anomalies: read() at 12.3ms (Z-score: 4.2)

# Real-time monitoring
renacer --realtime-anomalies -- ./myapp
# Live alerts for outliers

Anomaly Types Detected:

Duration anomalies (slow syscalls)
Frequency anomalies (unusual call patterns)
Sequential anomalies (unexpected syscall sequences)

Documentation: Anomaly Detection, Post-Hoc Analysis, Real-Time Monitoring chapters

Sprint 19: Enhanced Statistics

Goal: Advanced statistical metrics (percentiles, tail latency)

Features:

Percentiles - p50, p90, p95, p99, p99.9 for tail latency analysis
Distribution Analysis - Min, max, mean, median, stddev
Per-Syscall Stats - Individual percentiles for each syscall type

Example:

renacer -c -- ./myapp
# Output:
# read: calls=1000, p50=1.2ms, p95=3.4ms, p99=8.7ms, max=45.2ms
# write: calls=500, p50=2.1ms, p95=5.6ms, p99=12.3ms, max=67.8ms

Statistical Methods:

Interpolation - Linear interpolation for fractional percentiles
Sorting - Efficient O(n log n) percentile calculation
Outlier Detection - IQR method (Q1 - 1.5×IQR, Q3 + 1.5×IQR)

Documentation: Percentile Analysis, SIMD Acceleration chapters

Sprint 18: Multi-Process Tracing

Goal: Trace parent and all child processes (fork/exec following)

Features:

Fork Following - -f flag to trace child processes
Per-Process Stats - Individual statistics for each PID
Process Tree - Visualize parent-child relationships

Example:

renacer -f -c -- make
# Output shows:
# [12345] openat(...) = 3
# [12346] execve("/bin/gcc", ...) = 0  ← child
# [12345] waitpid(12346, ...) = 12346

Technical Details:

PTRACE_O_TRACEFORK - Automatically attach to forked children
PTRACE_O_TRACEEXEC - Trace exec() calls
Per-process DWARF correlation

Documentation: Multi-Process Tracing chapter

Sprint 16: Regex Pattern Matching

Goal: Powerful regex-based syscall filtering

Features:

Regex Syntax - /pattern/ for regular expressions
Pattern Examples:
- /^open.*/ - Match syscalls starting with "open"
- /.*at$/ - Match syscalls ending with "at"
- /read|write/ - OR operator
- /(?i)OPEN/ - Case-insensitive
Mixed Filtering - Combine regex, literals, negation, classes

Examples:

# Prefix matching
renacer -e 'trace=/^open.*/' -- ls

# Suffix matching
renacer -e 'trace=/.*at$/' -- ls

# OR operator
renacer -e 'trace=/read|write/' -- app

# Complex mix
renacer -e 'trace=file,!/openat/,/^fstat/' -- ls

RED-GREEN-REFACTOR:

RED: 9 integration tests (7 failing initially)
GREEN: Implemented regex parser, Regex crate integration
REFACTOR: 14 unit tests, complexity ≤10, zero clippy warnings

Version: 0.3.0-dev Test Count: 201 tests (178 + 23 new)

Sprint 15: Negation Operator

Goal: Exclude specific syscalls from tracing

Features:

Negation Syntax - !syscall to exclude
Mixed Filters - Combine include/exclude: trace=file,!openat
Class Negation - Exclude entire classes: trace=!network

Examples:

# Exclude specific syscalls
renacer -e 'trace=file,!openat' -- ls

# Trace everything except read/write
renacer -e 'trace=!read,!write' -- app

# Class with negation
renacer -e 'trace=file,!openat,!close' -- ls

Technical Details:

Parser updates: detect ! prefix
Filter logic: exclude takes precedence
Works with literals, classes, regex (Sprint 16)

Version: 0.2.5-dev Test Count: 178 tests

Sprint 14: Syscall Classes

Goal: Predefined groups of related syscalls

Features:

Classes Defined:
- file - File operations (open, read, write, close, stat, etc.)
- network - Network syscalls (socket, bind, connect, send, recv, etc.)
- ipc - Inter-process communication (pipe, mmap, shmget, etc.)
- desc - Descriptor operations (dup, fcntl, ioctl, etc.)
- process - Process management (fork, exec, wait, exit, etc.)
- signal - Signal handling (kill, sigaction, etc.)

Examples:

# Trace all file operations
renacer -e trace=file -- ls

# Trace network syscalls
renacer -e trace=network -- curl https://example.com

# Mix classes and literals
renacer -e trace=file,network,mmap -- ./myapp

Technical Details:

Class definitions in src/filter.rs
Expansion at parse time (class → syscall list)
Combinable with negation (Sprint 15) and regex (Sprint 16)

Version: 0.2.0-dev Test Count: ~160 tests

Sprint 13: Function Profiling

Goal: Attribute syscall execution time to source functions using DWARF

Features:

Function-Level Profiling - --function-time flag
DWARF Correlation - Map syscalls to functions via debug info
I/O Bottleneck Detection - Identify slow functions (>1ms threshold)
Stack Unwinding - Frame pointer chain walking (max 64 frames)

Example:

renacer --function-time -- ./myapp

# Output:
# Function: read_config (config.c:42)
#   openat: 2.3ms
#   read: 45.8ms       ← Bottleneck!
#   close: 0.1ms
#   Total: 48.2ms

Technical Details:

Requires -g (debug symbols) and -fno-omit-frame-pointer
DWARF parsing with gimli crate
Frame pointer unwinding (rbp register on x86_64)

Documentation: Function Profiling, I/O Bottleneck Detection, Call Graph Analysis chapters

Version: 0.1.5-dev Test Count: ~150 tests

Version 0.1.0 - Sprints 11-12

Release Date: 2025-10-28 Theme: Foundation & Quality Infrastructure

Sprint 11-12: Test Coverage & Benchmarks

Goal: Establish comprehensive testing and performance baseline

Features:

llvm-cov Coverage - HTML reports, LCOV export, 85%+ coverage
Benchmark Suite - Comprehensive tests vs strace
Makefile - Professional build automation
Property-Based Testing - 18 comprehensive proptest tests
Quality Gates - Pre-commit hooks with format, clippy, property tests, security audit

Benchmark Results:

Workload	strace	Renacer	Overhead
File I/O (1000 r/w)	0.18s	0.21s	1.17×
Syscall-heavy (10K)	0.22s	0.25s	1.14×
CPU-bound	0.05s	0.054s	1.08×

Quality Metrics:

Coverage: 87.3% (target: 85%)
Mutation Testing: cargo-mutants integration
Complexity: All functions ≤10
TDG Score: 92.0/100

Documentation:

Performance Benchmarks chapter
EXTREME TDD methodology
RED-GREEN-REFACTOR workflow

Version: 0.1.0 Test Count: ~140 tests

Sprint Numbering

Note: Sprint numbering is non-sequential to align with parallel projects (trueno, aprender, decy).

Sprints 11-12: Foundation (coverage, benchmarks)
Sprint 13: Function profiling
Sprint 14: Syscall classes
Sprint 15: Negation operator
Sprint 16: Regex patterns
Sprint 18: Multi-process tracing (Sprint 17 was trueno)
Sprint 19: Enhanced statistics
Sprint 20: Anomaly detection
Sprint 21: HPU acceleration
Sprint 22: HTML output
Sprint 23: ML anomaly detection
Sprints 24-28: Transpiler source mapping (5 phases)
Sprint 29: Chaos engineering & fuzz testing

Quality Metrics Progression

Sprint	Tests	Coverage	TDG Score	Max Complexity
11-12	140	87.3%	92.0	10
13	150	88.5%	93.2	10
14	160	89.1%	93.8	10
15	178	90.2%	94.0	10
16	201	91.5%	94.5	8
18	205	91.8%	94.5	10
19	210	92.1%	94.5	10
20	215	92.5%	94.5	10
21	220	92.8%	94.5	10
22	210	92.3%	94.5	10
23	220	92.7%	94.5	10
29	201	91.5%	94.5	10

Consistency Highlights:

✅ All sprints maintain ≤10 complexity (EXTREME TDD requirement)
✅ Coverage steadily increases (87% → 92%)
✅ TDG Score remains 94.0-94.5 (excellent quality)
✅ Zero regressions in quality metrics

Toyota Way Principles Applied

Throughout all sprints, Renacer follows Toyota Production System principles:

STOP THE LINE - Sprint 29: Eliminated flaky test immediately (zero-tolerance for defects)
Kaizen - Continuous improvement: complexity reduction, test coverage increase
Jidoka - Built-in quality: pre-commit hooks, mutation testing, property-based testing
Respect for People - Clear documentation, comprehensive examples, zero hallucination
Long-Term Philosophy - Sustainable pace, technical debt paydown

Hansei (Reflection): After each sprint, retrospective analysis identifies improvements for next sprint.

Future Roadmap

Planned Features

Sprint 30: Differential Testing (Oracle Problem)

Compare Renacer output against strace (ground truth)
Automated regression detection
Compatibility verification

Sprint 31: Call Graph Visualization

Export to Graphviz DOT format
Interactive call graphs
Flamegraph integration

Sprint 32: Advanced Filtering Operators

AND operator: trace=file&network (syscalls in both classes)
XOR operator: trace=file^network (exclusive or)
Parentheses: trace=(file|network)&!openat

Sprint 33: Container Awareness

Docker/Podman integration
Namespace-aware tracing
cgroup correlation

Sprint 34: eBPF Integration

Lower overhead vs ptrace
Kernel-level tracing
BPF CO-RE support

Glossary - Technical terms and definitions
FAQ - Frequently asked questions
Performance Tables - Detailed benchmark data
Benchmarks - Performance comparison vs strace

The Renacer Book - Pure Rust System Call Tracing