Debug Compilation & Transpilation - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# Example: Debug Compilation and Transpilation

This example demonstrates debugging the entire compilation pipeline: from Python source code through transpilation (Depyler), compilation (rustc), to final binary execution. Perfect for diagnosing issues like DEPYLER-0435.

## Overview: The Full Pipeline

When transpiling Python to Rust with Depyler, we have multiple layers:

    Python Source (.py)
        ↓ [Depyler Transpiler]
    Generated Rust (.rs) + Source Map (.map)
        ↓ [cargo/rustc Compiler]
    Binary Executable + DWARF Debug Info
        ↓ [Execution]
    Runtime Syscalls

Renacer can trace and correlate across ALL these layers.

## Scenario 1: Debugging Slow Compilation

Your Depyler-transpiled project takes 15 minutes to compile. Let's find why.

### Step 1: Measure Compilation Performance

    # Baseline: How long does it take?
    $ time cargo build --release

    real    15m23s
    user    42m15s
    sys     2m34s

Problem: 15 minutes is too slow. What's happening?

### Step 2: Trace the Build Process

    $ renacer -f -c -e 'trace=file,process' -- cargo build --release

Output (summary):

    System Call Summary (Multi-Process):
    =====================================
    Process Tree:
    [PID 12345] cargo build
      └─ [PID 12346] rustc --crate-name myapp ...
           ├─ [PID 12347] rustc --crate-name dep1 ...
           ├─ [PID 12348] rustc --crate-name dep2 ...
           └─ [PID 12349] ld (linker)

    Syscall Statistics:
    Syscall    Total Calls   Total Time    Process Distribution
    openat     125,847       45.2s         rustc: 98%, ld: 2%
    read       2,456,789     23.4s         rustc: 100%
    write      345,678       12.3s         rustc: 95%, ld: 5%
    execve     1,247         8.9s          cargo: 100%

Analysis:

- 125,847 `openat` calls taking 45 seconds
- 1,247 `execve` calls (many rustc invocations)
- Most time in `rustc` reading files

### Step 3: Find What's Being Opened

    $ renacer -f -e 'trace=openat' -- cargo build --release 2>&1 | \
      grep -o '"/[^"]*"' | sort | uniq -c | sort -rn | head -20

Output:

    12,456 "/home/user/.cargo/registry/cache/..."
    8,923  "/usr/lib/rustlib/x86_64-unknown-linux-gnu/..."
    4,567  "/home/user/project/target/release/deps/..."
    3,421  "/home/user/project/src/generated/mod_0001.rs"
    3,420  "/home/user/project/src/generated/mod_0002.rs"
    3,419  "/home/user/project/src/generated/mod_0003.rs"
    ... (1,200 more generated modules)

Root Cause Found: Depyler generated 1,200+ separate `.rs` modules, each opened ~3 times during compilation!

### Step 4: Correlate with Source Code

    # Find where rustc is spending time
    $ renacer -f --source -e 'trace=read' -- cargo build --release 2>&1 | \
      grep "rustc" | head -10

Output:

    [PID 12346] read(3, "// Generated by Depyler\nmod mo...", 8192) = 8192
      [/rustc/hash/compiler/rustc_parse/src/parser/item.rs:234 in parse_item]

    [PID 12346] read(4, "// Generated by Depyler\npub fn...", 8192) = 8192
      [/rustc/hash/compiler/rustc_parse/src/parser/item.rs:234 in parse_item]

Insight: `rustc` parser is reading each generated file individually.

### Step 5: Analyze Depyler Output Structure

    # Trace the transpilation step
    $ renacer -e 'trace=file' -- depyler transpile main.py --output src/generated/

Output:

    openat(AT_FDCWD, "main.py", O_RDONLY) = 3
    # ... parsing ...
    openat(AT_FDCWD, "src/generated/mod_0001.rs", O_WRONLY|O_CREAT) = 4
    write(4, "// Generated by Depyler\n...", 2456) = 2456
    close(4) = 0
    openat(AT_FDCWD, "src/generated/mod_0002.rs", O_WRONLY|O_CREAT) = 4
    write(4, "// Generated by Depyler\n...", 2398) = 2398
    close(4) = 0
    # ... 1,198 more files ...

Problem: Depyler is generating 1,200 small modules instead of consolidating them.

Solution: Configure Depyler to use `--merge-modules` flag:

    $ depyler transpile main.py --output src/generated/ --merge-modules=100

Result: 1,200 modules → 12 modules (100 items per file)

### Step 6: Verify Improvement

    $ time cargo build --release

    real    2m34s   ← 6x faster!
    user    7m12s
    sys     0m23s

Improvement:

- Compilation time: 15m23s → 2m34s (6x faster)
- `openat` calls: 125,847 → 12,456 (10x reduction)

## Scenario 2: Finding Missing Dependencies

Your transpiled code fails to compile with cryptic error. Let's trace it.

### Problem Statement

    $ cargo build
       Compiling myapp v0.1.0
    error[E0433]: failed to resolve: use of undeclared crate or module `std`
      --> src/generated/mod_main.rs:42:5
       |
    42 |     std::fs::read_to_string(path)
       |     ^^^ use of undeclared crate or module `std`

Question: Why can't it find `std`? It should be automatically available!

### Step 1: Trace the Failing Compilation

    $ renacer -f --source -e 'trace=openat' -- cargo build 2>&1 | \
      grep -E "\.rs|ENOENT"

Output:

    [PID 12350] openat(AT_FDCWD, "src/generated/mod_main.rs", O_RDONLY) = 3
    [PID 12350] openat(AT_FDCWD, "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib", O_RDONLY) = 4
    [PID 12350] openat(AT_FDCWD, "src/generated/prelude.rs", O_RDONLY) = -2 (ENOENT)
    [PID 12350] openat(AT_FDCWD, "src/lib.rs", O_RDONLY) = -2 (ENOENT)

Analysis:

- `rustc` successfully opens `libstd.rlib` (so std EXISTS)
- Looking for `prelude.rs` but not found (ENOENT)
- The generated code is missing the prelude import!

### Step 2: Check Generated Code

    $ head -20 src/generated/mod_main.rs

Output:

    // Generated by Depyler v0.4.0
    // Source: main.py

    #![no_std]  ← PROBLEM! This disables std library

    pub fn read_file(path: &str) -> String {
        std::fs::read_to_string(path)  // ← Tries to use std, but no_std is set!
    }

Root Cause: Depyler incorrectly set `#![no_std]` attribute in generated code.

### Step 3: Trace Depyler Transpilation

    $ renacer --source -e 'trace=write' -- depyler transpile main.py 2>&1 | \
      grep "no_std"

Output:

    write(4, "#![no_std]\n", 11) = 11
      [depyler/src/codegen/rust_emitter.rs:156 in emit_crate_attrs]

Found It: `depyler/src/codegen/rust_emitter.rs:156` is emitting `#![no_std]` incorrectly.

### Step 4: Fix and Verify

Edit Depyler configuration:

    # depyler.toml
    [codegen]
    use_std = true  # ← Force std library usage
    no_std = false

Recompile:

    $ depyler transpile main.py
    $ cargo build
       Compiling myapp v0.1.0
       Finished dev [unoptimized + debuginfo] target(s) in 12.3s

Success!

## Scenario 3: Transpiler Source Mapping (Sprint 24-28)

A runtime error occurs in transpiled code. Can we trace it back to the original Python?

### Problem: Runtime Panic in Generated Rust

    $ ./target/release/myapp
    thread 'main' panicked at 'index out of bounds: the len is 5 but the index is 10',
      src/generated/mod_array_ops.rs:234:9

Question: What line in the original Python caused this?

### Step 1: Trace Runtime Execution

    $ renacer --source -- ./target/release/myapp

Output:

    read(3, "[1, 2, 3, 4, 5]", 4096) = 15
      [src/generated/mod_array_ops.rs:230 in parse_array]

    read(3, "10", 4096) = 2
      [src/generated/mod_array_ops.rs:232 in read_index]

    # PANIC at src/generated/mod_array_ops.rs:234

DWARF tells us: Crash is at `mod_array_ops.rs:234`

But we need: Original Python source location!

### Step 2: Use Transpiler Source Map

    $ renacer --source --transpiler-map src/generated.map -- ./target/release/myapp

Output (with source map):

    read(3, "[1, 2, 3, 4, 5]", 4096) = 15
      [src/generated/mod_array_ops.rs:230 in parse_array]
      ↳ [Original: main.py:42 in process_data]  ← Transpiler mapping!

    read(3, "10", 4096) = 2
      [src/generated/mod_array_ops.rs:232 in read_index]
      ↳ [Original: main.py:43 in process_data]

    # PANIC at src/generated/mod_array_ops.rs:234
      ↳ [Original: main.py:44 in process_data]  ← Root cause in Python!

Correlation Chain:

    Python: main.py:44
       ↓ (transpiled to)
    Rust: src/generated/mod_array_ops.rs:234
       ↓ (compiled to)
    Binary: 0x401234 (with DWARF)
       ↓ (executes)
    Syscall: read(3, ...) = 2

### Step 3: Inspect Original Python Source

    # main.py:42-44 (the actual bug)
    def process_data(filename):
        data = json.load(open(filename))  # Line 42
        index = int(input("Enter index: "))  # Line 43
        return data[index]  # Line 44 ← Out of bounds!

Root Cause Found: Python code doesn't validate `index` before accessing `data[index]`.

Fix:

    def process_data(filename):
        data = json.load(open(filename))
        index = int(input("Enter index: "))
        if index < 0 or index >= len(data):
            raise IndexError(f"Index {index} out of range [0, {len(data)})")
        return data[index]

## Scenario 4: Multi-Process Compilation Pipeline

Understanding the full build dependency tree.

### Step 1: Trace Full Build with Process Tree

    $ renacer -f -c --function-time -- cargo build

Output:

    Process Hierarchy:
    ==================
    [PID 10001] cargo build (parent)
        Time in execve: 234ms (spawning children)
        Time in wait4: 145.2s (waiting for rustc)

      [PID 10002] rustc --crate-name myapp (child 1)
          Time in openat: 12.3s (reading source files)
          Time in read: 45.6s (parsing)
          Time in write: 8.9s (emitting LLVM IR)
          Time in execve: 1.2s (spawning linker)

        [PID 10003] ld (linker, grandchild)
            Time in openat: 3.4s (reading .o files)
            Time in read: 8.9s (linking)
            Time in write: 2.1s (writing binary)

      [PID 10004] rustc --crate-name dep1 (child 2 - parallel)
          Time in openat: 5.6s
          Time in read: 23.4s
          Time in write: 4.3s

      [PID 10005] rustc --crate-name dep2 (child 3 - parallel)
          Time in openat: 4.2s
          Time in read: 18.9s
          Time in write: 3.1s

Analysis:

- `cargo` spends 145s waiting for child processes
- 3 `rustc` processes run in parallel
- Each `rustc` spends most time in `read` (parsing)
- Linker (`ld`) is relatively fast (14.4s total)

### Step 2: Find Slowest Compilation Unit

    $ renacer -f -c -e 'trace=file' -- cargo build 2>&1 | \
      grep -A 3 "rustc.*myapp"

Output:

    [PID 10002] rustc --crate-name myapp
      Syscall Statistics:
      openat: 8,923 calls, 12.3s total
      read:   125,847 calls, 45.6s total  ← Slowest operation
      write:  34,521 calls, 8.9s total

Bottleneck: `rustc` reading 125,847 times (parsing generated code)

### Step 3: Optimize Build Parallelism

    # Before: Serial compilation
    $ time cargo build -j 1
    real    8m34s

    # After: Parallel compilation
    $ time cargo build -j 4
    real    2m45s  ← 3x faster

Result: Multi-process tracing revealed parallelization opportunity.

## Scenario 5: Debugging Compilation Errors

The compiler fails with an error. Where is the problem in the pipeline?

### Problem: Mysterious Type Error

    $ cargo build
    error[E0308]: mismatched types
      --> src/generated/mod_types.rs:89:5
       |
    89 |     result
       |     ^^^^^^ expected `i32`, found `String`

### Step 1: Trace Code Generation

    $ renacer --transpiler-map depyler.map -- depyler transpile types.py

Output:

    write(4, "fn convert(x: String) -> i32 {\n", 31) = 31
      [depyler/src/codegen/type_inference.rs:123 in emit_function]
      ↳ [Original: types.py:12 in convert]

    write(4, "    result\n", 11) = 11
      [depyler/src/codegen/type_inference.rs:145 in emit_return]
      ↳ [Original: types.py:15 in convert]  ← Wrong type inferred!

Root Cause: Depyler's type inference at `type_inference.rs:123` incorrectly assumed `String` when it should be `i32`.

### Step 2: Check Original Python Type Hint

    # types.py:12-15
    def convert(x: str) -> int:  # ← Type hint says int
        result = int(x)  # Line 14
        return result  # Line 15 ← Should return int, not str

Problem: Depyler didn't respect Python type hints during code generation.

### Step 3: Fix Depyler Type Mapping

Report bug to Depyler:

- Type hint `int` should map to Rust `i32`
- Type inference failed to see `int(x)` converts to integer

## Advanced: Full Stack Correlation

Combining all features to trace from Python source to binary execution.

### Scenario: Performance Regression After Transpilation

Python version runs fast, but transpiled Rust version is slow. Why?

### Step 1: Compare Python vs Transpiled Performance

    # Python baseline
    $ time python3 main.py
    real    0.5s

    # Transpiled Rust version
    $ time ./target/release/myapp
    real    5.2s  ← 10x slower!

### Step 2: Profile Both Versions

    # Profile Python
    $ renacer -c -e 'trace=file' -- python3 main.py

Output (Python):

    Syscall Statistics:
    openat: 12 calls, 23ms
    read:   156 calls, 45ms
    write:  89 calls, 12ms
    Total:  80ms (rest is Python interpreter overhead)

    # Profile Transpiled Rust
    $ renacer -c -e 'trace=file' -- ./target/release/myapp

Output (Rust):

    Syscall Statistics:
    openat: 1,247 calls, 2.3s  ← 100x more opens!
    read:   156 calls, 45ms
    write:  89 calls, 12ms
    Total:  2.4s

Problem: Rust version makes 1,247 `openat` calls vs 12 in Python!

### Step 3: Find Source of Extra Opens

    $ renacer --source --transpiler-map depyler.map -e 'trace=openat' -- ./target/release/myapp 2>&1 | head -50

Output:

    openat(AT_FDCWD, "config.json", O_RDONLY) = 3
      [src/generated/mod_config.rs:45 in load_config]
      ↳ [Original: main.py:23 in initialize]

    openat(AT_FDCWD, "config.json", O_RDONLY) = 3
      [src/generated/mod_config.rs:45 in load_config]
      ↳ [Original: main.py:23 in initialize]  ← Same file opened again!

    # ... 1,245 more identical opens ...

Root Cause: Transpiled code calls `load_config()` in a loop without caching!

### Step 4: Check Original Python

    # main.py:22-30
    def initialize():
        for i in range(1000):
            config = load_config()  # Line 23 ← Opens file every iteration!
            process_item(i, config)

Problem: Python's file caching masked this issue, but Rust opens file every time.

Fix:

    def initialize():
        config = load_config()  # ← Cache outside loop
        for i in range(1000):
            process_item(i, config)

Result After Fix:

    $ time ./target/release/myapp
    real    0.4s  ← 13x faster! Even faster than Python!

## Debugging Workflow Summary

### Full Pipeline Debugging Process:

    # 1. Trace transpilation (Depyler)
    $ renacer -e 'trace=file' -- depyler transpile main.py

    # 2. Trace compilation (cargo/rustc)
    $ renacer -f -c -e 'trace=file,process' -- cargo build

    # 3. Trace execution with source maps
    $ renacer --source --transpiler-map out.map -- ./myapp

    # 4. Export for analysis
    $ renacer --format json --transpiler-map out.map -- ./myapp > trace.json

### Key Techniques:

1. `-f` (follow forks) - Trace multi-process builds
2. `--source` (DWARF) - Correlate binary → Rust source
3. `--transpiler-map` - Correlate Rust → Python source
4. `-e trace=file,process` - Focus on build-relevant syscalls
5. `-c` (statistics) - Measure compilation performance

## Common Compilation Issues

### Issue 1: Missing Source Maps

Symptom:

    $ renacer --transpiler-map out.map -- ./myapp
    Error: Transpiler map not found: out.map

Solution:

    # Ensure Depyler generates source maps
    $ depyler transpile main.py --emit-source-map
    # Creates: src/generated.map

### Issue 2: DWARF Info Stripped

Symptom:

    $ renacer --source -- ./target/release/myapp
    Warning: No DWARF debug info found

Solution:

    # Always compile with debug info
    $ cargo build --release  # Includes debug info by default
    # OR explicitly:
    $ RUSTFLAGS="-C debuginfo=2" cargo build --release

### Issue 3: Too Many Processes to Track

Symptom:

    $ renacer -f -- cargo build
    Error: Process limit exceeded (ptrace: ENOMEM)

Solution:

    # Limit parallel jobs
    $ renacer -f -- cargo build -j 4

    # OR filter to specific crate
    $ renacer -f -- cargo build -p myapp

## Best Practices

### 1. Always Use Source Maps for Transpiled Code

    # Generate with source maps
    $ depyler transpile --emit-source-map main.py

    # Trace with source maps
    $ renacer --transpiler-map out.map -- ./myapp

### 2. Profile Before Optimizing

    # Baseline first
    $ renacer -c -- cargo build > before.txt

    # Optimize, then compare
    $ renacer -c -- cargo build > after.txt
    $ diff before.txt after.txt

### 3. Use Multi-Process Tracing for Builds

    # Always use -f for build systems
    $ renacer -f -c -- cargo build
    $ renacer -f -c -- make

### 4. Export for CI/CD Integration

    # Export compilation stats for regression detection
    $ renacer -f --format json -c -- cargo build > build-trace.json

    # CI script checks:
    # - Total compilation time < threshold
    # - No excessive file opens during linking
    # - Process tree depth reasonable

## Related

- Multi-Process Tracing - Using `-f` flag
- DWARF Source Correlation - `--source` flag
- Export to JSON/CSV - Automated analysis
- Debug Performance Issues - General profiling