Example: Debug Compilation and Transpilation

This example demonstrates debugging the entire compilation pipeline: from Python source code through transpilation (Depyler), compilation (rustc), to final binary execution. Perfect for diagnosing issues like DEPYLER-0435.

Overview: The Full Pipeline

When transpiling Python to Rust with Depyler, we have multiple layers:

Python Source (.py)
    ↓ [Depyler Transpiler]
Generated Rust (.rs) + Source Map (.map)
    ↓ [cargo/rustc Compiler]
Binary Executable + DWARF Debug Info
    ↓ [Execution]
Runtime Syscalls

Renacer can trace and correlate across ALL these layers.

Scenario 1: Debugging Slow Compilation

Your Depyler-transpiled project takes 15 minutes to compile. Let's find why.

Step 1: Measure Compilation Performance

# Baseline: How long does it take?
$ time cargo build --release

real    15m23s
user    42m15s
sys     2m34s

Problem: 15 minutes is too slow. What's happening?

Step 2: Trace the Build Process

$ renacer -f -c -e 'trace=file,process' -- cargo build --release

Output (summary):

System Call Summary (Multi-Process):
=====================================
Process Tree:
[PID 12345] cargo build
  └─ [PID 12346] rustc --crate-name myapp ...
       ├─ [PID 12347] rustc --crate-name dep1 ...
       ├─ [PID 12348] rustc --crate-name dep2 ...
       └─ [PID 12349] ld (linker)

Syscall Statistics:
Syscall    Total Calls   Total Time    Process Distribution
openat     125,847       45.2s         rustc: 98%, ld: 2%
read       2,456,789     23.4s         rustc: 100%
write      345,678       12.3s         rustc: 95%, ld: 5%
execve     1,247         8.9s          cargo: 100%

Analysis:

  • 125,847 openat calls taking 45 seconds
  • 1,247 execve calls (many rustc invocations)
  • Most time in rustc reading files

Step 3: Find What's Being Opened

$ renacer -f -e 'trace=openat' -- cargo build --release 2>&1 | \
  grep -o '"/[^"]*"' | sort | uniq -c | sort -rn | head -20

Output:

12,456 "/home/user/.cargo/registry/cache/..."
8,923  "/usr/lib/rustlib/x86_64-unknown-linux-gnu/..."
4,567  "/home/user/project/target/release/deps/..."
3,421  "/home/user/project/src/generated/mod_0001.rs"
3,420  "/home/user/project/src/generated/mod_0002.rs"
3,419  "/home/user/project/src/generated/mod_0003.rs"
... (1,200 more generated modules)

Root Cause Found: Depyler generated 1,200+ separate .rs modules, each opened ~3 times during compilation!

Step 4: Correlate with Source Code

# Find where rustc is spending time
$ renacer -f --source -e 'trace=read' -- cargo build --release 2>&1 | \
  grep "rustc" | head -10

Output:

[PID 12346] read(3, "// Generated by Depyler\nmod mo...", 8192) = 8192
  [/rustc/hash/compiler/rustc_parse/src/parser/item.rs:234 in parse_item]

[PID 12346] read(4, "// Generated by Depyler\npub fn...", 8192) = 8192
  [/rustc/hash/compiler/rustc_parse/src/parser/item.rs:234 in parse_item]

Insight: rustc parser is reading each generated file individually.

Step 5: Analyze Depyler Output Structure

# Trace the transpilation step
$ renacer -e 'trace=file' -- depyler transpile main.py --output src/generated/

Output:

openat(AT_FDCWD, "main.py", O_RDONLY) = 3
# ... parsing ...
openat(AT_FDCWD, "src/generated/mod_0001.rs", O_WRONLY|O_CREAT) = 4
write(4, "// Generated by Depyler\n...", 2456) = 2456
close(4) = 0
openat(AT_FDCWD, "src/generated/mod_0002.rs", O_WRONLY|O_CREAT) = 4
write(4, "// Generated by Depyler\n...", 2398) = 2398
close(4) = 0
# ... 1,198 more files ...

Problem: Depyler is generating 1,200 small modules instead of consolidating them.

Solution: Configure Depyler to use --merge-modules flag:

$ depyler transpile main.py --output src/generated/ --merge-modules=100

Result: 1,200 modules → 12 modules (100 items per file)

Step 6: Verify Improvement

$ time cargo build --release

real    2m34s   ← 6x faster!
user    7m12s
sys     0m23s

Improvement:

  • Compilation time: 15m23s → 2m34s (6x faster)
  • openat calls: 125,847 → 12,456 (10x reduction)

Scenario 2: Finding Missing Dependencies

Your transpiled code fails to compile with cryptic error. Let's trace it.

Problem Statement

$ cargo build
   Compiling myapp v0.1.0
error[E0433]: failed to resolve: use of undeclared crate or module `std`
  --> src/generated/mod_main.rs:42:5
   |
42 |     std::fs::read_to_string(path)
   |     ^^^ use of undeclared crate or module `std`

Question: Why can't it find std? It should be automatically available!

Step 1: Trace the Failing Compilation

$ renacer -f --source -e 'trace=openat' -- cargo build 2>&1 | \
  grep -E "\.rs|ENOENT"

Output:

[PID 12350] openat(AT_FDCWD, "src/generated/mod_main.rs", O_RDONLY) = 3
[PID 12350] openat(AT_FDCWD, "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib", O_RDONLY) = 4
[PID 12350] openat(AT_FDCWD, "src/generated/prelude.rs", O_RDONLY) = -2 (ENOENT)
[PID 12350] openat(AT_FDCWD, "src/lib.rs", O_RDONLY) = -2 (ENOENT)

Analysis:

  • rustc successfully opens libstd.rlib (so std EXISTS)
  • Looking for prelude.rs but not found (ENOENT)
  • The generated code is missing the prelude import!

Step 2: Check Generated Code

$ head -20 src/generated/mod_main.rs

Output:

// Generated by Depyler v0.4.0
// Source: main.py

#![no_std]  ← PROBLEM! This disables std library

pub fn read_file(path: &str) -> String {
    std::fs::read_to_string(path)  // ← Tries to use std, but no_std is set!
}

Root Cause: Depyler incorrectly set #![no_std] attribute in generated code.

Step 3: Trace Depyler Transpilation

$ renacer --source -e 'trace=write' -- depyler transpile main.py 2>&1 | \
  grep "no_std"

Output:

write(4, "#![no_std]\n", 11) = 11
  [depyler/src/codegen/rust_emitter.rs:156 in emit_crate_attrs]

Found It: depyler/src/codegen/rust_emitter.rs:156 is emitting #![no_std] incorrectly.

Step 4: Fix and Verify

Edit Depyler configuration:

# depyler.toml
[codegen]
use_std = true  # ← Force std library usage
no_std = false

Recompile:

$ depyler transpile main.py
$ cargo build
   Compiling myapp v0.1.0
   Finished dev [unoptimized + debuginfo] target(s) in 12.3s

Success!

Scenario 3: Transpiler Source Mapping (Sprint 24-28)

A runtime error occurs in transpiled code. Can we trace it back to the original Python?

Problem: Runtime Panic in Generated Rust

$ ./target/release/myapp
thread 'main' panicked at 'index out of bounds: the len is 5 but the index is 10',
  src/generated/mod_array_ops.rs:234:9

Question: What line in the original Python caused this?

Step 1: Trace Runtime Execution

$ renacer --source -- ./target/release/myapp

Output:

read(3, "[1, 2, 3, 4, 5]", 4096) = 15
  [src/generated/mod_array_ops.rs:230 in parse_array]

read(3, "10", 4096) = 2
  [src/generated/mod_array_ops.rs:232 in read_index]

# PANIC at src/generated/mod_array_ops.rs:234

DWARF tells us: Crash is at mod_array_ops.rs:234

But we need: Original Python source location!

Step 2: Use Transpiler Source Map

$ renacer --source --transpiler-map src/generated.map -- ./target/release/myapp

Output (with source map):

read(3, "[1, 2, 3, 4, 5]", 4096) = 15
  [src/generated/mod_array_ops.rs:230 in parse_array]
  ↳ [Original: main.py:42 in process_data]  ← Transpiler mapping!

read(3, "10", 4096) = 2
  [src/generated/mod_array_ops.rs:232 in read_index]
  ↳ [Original: main.py:43 in process_data]

# PANIC at src/generated/mod_array_ops.rs:234
  ↳ [Original: main.py:44 in process_data]  ← Root cause in Python!

Correlation Chain:

Python: main.py:44
   ↓ (transpiled to)
Rust: src/generated/mod_array_ops.rs:234
   ↓ (compiled to)
Binary: 0x401234 (with DWARF)
   ↓ (executes)
Syscall: read(3, ...) = 2

Step 3: Inspect Original Python Source

# main.py:42-44 (the actual bug)
def process_data(filename):
    data = json.load(open(filename))  # Line 42
    index = int(input("Enter index: "))  # Line 43
    return data[index]  # Line 44 ← Out of bounds!

Root Cause Found: Python code doesn't validate index before accessing data[index].

Fix:

def process_data(filename):
    data = json.load(open(filename))
    index = int(input("Enter index: "))
    if index < 0 or index >= len(data):
        raise IndexError(f"Index {index} out of range [0, {len(data)})")
    return data[index]

Scenario 4: Multi-Process Compilation Pipeline

Understanding the full build dependency tree.

Step 1: Trace Full Build with Process Tree

$ renacer -f -c --function-time -- cargo build

Output:

Process Hierarchy:
==================
[PID 10001] cargo build (parent)
    Time in execve: 234ms (spawning children)
    Time in wait4: 145.2s (waiting for rustc)

  [PID 10002] rustc --crate-name myapp (child 1)
      Time in openat: 12.3s (reading source files)
      Time in read: 45.6s (parsing)
      Time in write: 8.9s (emitting LLVM IR)
      Time in execve: 1.2s (spawning linker)

    [PID 10003] ld (linker, grandchild)
        Time in openat: 3.4s (reading .o files)
        Time in read: 8.9s (linking)
        Time in write: 2.1s (writing binary)

  [PID 10004] rustc --crate-name dep1 (child 2 - parallel)
      Time in openat: 5.6s
      Time in read: 23.4s
      Time in write: 4.3s

  [PID 10005] rustc --crate-name dep2 (child 3 - parallel)
      Time in openat: 4.2s
      Time in read: 18.9s
      Time in write: 3.1s

Analysis:

  • cargo spends 145s waiting for child processes
  • 3 rustc processes run in parallel
  • Each rustc spends most time in read (parsing)
  • Linker (ld) is relatively fast (14.4s total)

Step 2: Find Slowest Compilation Unit

$ renacer -f -c -e 'trace=file' -- cargo build 2>&1 | \
  grep -A 3 "rustc.*myapp"

Output:

[PID 10002] rustc --crate-name myapp
  Syscall Statistics:
  openat: 8,923 calls, 12.3s total
  read:   125,847 calls, 45.6s total  ← Slowest operation
  write:  34,521 calls, 8.9s total

Bottleneck: rustc reading 125,847 times (parsing generated code)

Step 3: Optimize Build Parallelism

# Before: Serial compilation
$ time cargo build -j 1
real    8m34s

# After: Parallel compilation
$ time cargo build -j 4
real    2m45s  ← 3x faster

Result: Multi-process tracing revealed parallelization opportunity.

Scenario 5: Debugging Compilation Errors

The compiler fails with an error. Where is the problem in the pipeline?

Problem: Mysterious Type Error

$ cargo build
error[E0308]: mismatched types
  --> src/generated/mod_types.rs:89:5
   |
89 |     result
   |     ^^^^^^ expected `i32`, found `String`

Step 1: Trace Code Generation

$ renacer --transpiler-map depyler.map -- depyler transpile types.py

Output:

write(4, "fn convert(x: String) -> i32 {\n", 31) = 31
  [depyler/src/codegen/type_inference.rs:123 in emit_function]
  ↳ [Original: types.py:12 in convert]

write(4, "    result\n", 11) = 11
  [depyler/src/codegen/type_inference.rs:145 in emit_return]
  ↳ [Original: types.py:15 in convert]  ← Wrong type inferred!

Root Cause: Depyler's type inference at type_inference.rs:123 incorrectly assumed String when it should be i32.

Step 2: Check Original Python Type Hint

# types.py:12-15
def convert(x: str) -> int:  # ← Type hint says int
    result = int(x)  # Line 14
    return result  # Line 15 ← Should return int, not str

Problem: Depyler didn't respect Python type hints during code generation.

Step 3: Fix Depyler Type Mapping

Report bug to Depyler:

  • Type hint int should map to Rust i32
  • Type inference failed to see int(x) converts to integer

Advanced: Full Stack Correlation

Combining all features to trace from Python source to binary execution.

Scenario: Performance Regression After Transpilation

Python version runs fast, but transpiled Rust version is slow. Why?

Step 1: Compare Python vs Transpiled Performance

# Python baseline
$ time python3 main.py
real    0.5s

# Transpiled Rust version
$ time ./target/release/myapp
real    5.2s  ← 10x slower!

Step 2: Profile Both Versions

# Profile Python
$ renacer -c -e 'trace=file' -- python3 main.py

Output (Python):

Syscall Statistics:
openat: 12 calls, 23ms
read:   156 calls, 45ms
write:  89 calls, 12ms
Total:  80ms (rest is Python interpreter overhead)

# Profile Transpiled Rust
$ renacer -c -e 'trace=file' -- ./target/release/myapp

Output (Rust):

Syscall Statistics:
openat: 1,247 calls, 2.3s  ← 100x more opens!
read:   156 calls, 45ms
write:  89 calls, 12ms
Total:  2.4s

Problem: Rust version makes 1,247 openat calls vs 12 in Python!

Step 3: Find Source of Extra Opens

$ renacer --source --transpiler-map depyler.map -e 'trace=openat' -- ./target/release/myapp 2>&1 | head -50

Output:

openat(AT_FDCWD, "config.json", O_RDONLY) = 3
  [src/generated/mod_config.rs:45 in load_config]
  ↳ [Original: main.py:23 in initialize]

openat(AT_FDCWD, "config.json", O_RDONLY) = 3
  [src/generated/mod_config.rs:45 in load_config]
  ↳ [Original: main.py:23 in initialize]  ← Same file opened again!

# ... 1,245 more identical opens ...

Root Cause: Transpiled code calls load_config() in a loop without caching!

Step 4: Check Original Python

# main.py:22-30
def initialize():
    for i in range(1000):
        config = load_config()  # Line 23 ← Opens file every iteration!
        process_item(i, config)

Problem: Python's file caching masked this issue, but Rust opens file every time.

Fix:

def initialize():
    config = load_config()  # ← Cache outside loop
    for i in range(1000):
        process_item(i, config)

Result After Fix:

$ time ./target/release/myapp
real    0.4s  ← 13x faster! Even faster than Python!

Debugging Workflow Summary

Full Pipeline Debugging Process:

# 1. Trace transpilation (Depyler)
$ renacer -e 'trace=file' -- depyler transpile main.py

# 2. Trace compilation (cargo/rustc)
$ renacer -f -c -e 'trace=file,process' -- cargo build

# 3. Trace execution with source maps
$ renacer --source --transpiler-map out.map -- ./myapp

# 4. Export for analysis
$ renacer --format json --transpiler-map out.map -- ./myapp > trace.json

Key Techniques:

  1. -f (follow forks) - Trace multi-process builds
  2. --source (DWARF) - Correlate binary → Rust source
  3. --transpiler-map - Correlate Rust → Python source
  4. -e trace=file,process - Focus on build-relevant syscalls
  5. -c (statistics) - Measure compilation performance

Common Compilation Issues

Issue 1: Missing Source Maps

Symptom:

$ renacer --transpiler-map out.map -- ./myapp
Error: Transpiler map not found: out.map

Solution:

# Ensure Depyler generates source maps
$ depyler transpile main.py --emit-source-map
# Creates: src/generated.map

Issue 2: DWARF Info Stripped

Symptom:

$ renacer --source -- ./target/release/myapp
Warning: No DWARF debug info found

Solution:

# Always compile with debug info
$ cargo build --release  # Includes debug info by default
# OR explicitly:
$ RUSTFLAGS="-C debuginfo=2" cargo build --release

Issue 3: Too Many Processes to Track

Symptom:

$ renacer -f -- cargo build
Error: Process limit exceeded (ptrace: ENOMEM)

Solution:

# Limit parallel jobs
$ renacer -f -- cargo build -j 4

# OR filter to specific crate
$ renacer -f -- cargo build -p myapp

Best Practices

1. Always Use Source Maps for Transpiled Code

# Generate with source maps
$ depyler transpile --emit-source-map main.py

# Trace with source maps
$ renacer --transpiler-map out.map -- ./myapp

2. Profile Before Optimizing

# Baseline first
$ renacer -c -- cargo build > before.txt

# Optimize, then compare
$ renacer -c -- cargo build > after.txt
$ diff before.txt after.txt

3. Use Multi-Process Tracing for Builds

# Always use -f for build systems
$ renacer -f -c -- cargo build
$ renacer -f -c -- make

4. Export for CI/CD Integration

# Export compilation stats for regression detection
$ renacer -f --format json -c -- cargo build > build-trace.json

# CI script checks:
# - Total compilation time < threshold
# - No excessive file opens during linking
# - Process tree depth reasonable