Multi-Process Tracing - The Renacer Book - Pure Rust System Call Tracing
Description: A comprehensive guide to renacer: syscall tracing, DWARF correlation, and performance analysis

# Multi-Process Tracing

Renacer can trace entire process trees (parent + children) using the `-f` flag, making it ideal for analyzing parallel builds, shell scripts, and applications that fork child processes.

> TDD-Verified: All examples validated by `tests/sprint18_multiprocess_tests.rs` (11+ integration tests)

## Overview

Multi-process tracing automatically follows all child processes created via:

- fork() - Traditional Unix process creation
- vfork() - Lightweight fork variant
- clone() - Linux process/thread creation
- Fork + exec() - Child process replacement

### Why Multi-Process Tracing?

Without `-f` (default):

    $ renacer -- make -j8
    # Only traces the `make` parent process
    # Child compiler processes are NOT traced

With `-f` flag:

    $ renacer -f -- make -j8
    # Traces `make` + all 8 compiler child processes
    # Complete view of parallel build behavior

## Basic Usage

### Enable Multi-Process Tracing

    renacer -f -- ./my-app

Tested by: `test_follow_forks_basic`

This enables automatic following of all forked child processes.

### Fork Tracking Output

    $ renacer -f -- ./fork-example

Example Output:

    [pid 1234] clone(CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD) = 1235
    [pid 1235] write(1, "child process\n", 14) = 14
    [pid 1234] wait4(1235, NULL, 0, NULL) = 1235
    [pid 1234] write(1, "parent process\n", 15) = 15

Tested by: `test_follow_forks_basic`

Key indicators:

- `[pid XXXX]` - Shows which process made each syscall
- `clone()` - Linux implementation of fork() (creates child process)
- Parent continues after fork, child runs in parallel

### Disabled by Default

Without `-f`, only the parent process is traced:

    $ renacer -- ./fork-example
    # Child syscalls NOT shown

Tested by: `test_follow_forks_disabled_by_default`

This ensures backward compatibility and minimal overhead for single-process programs.

## Fork + Exec Pattern

Many programs fork then immediately exec a new program:

    $ renacer -f -- sh -c "ls /tmp"

Tested by: `test_follow_forks_with_exec`

Example Output:

    [pid 1234] clone(...) = 1235
    [pid 1235] execve("/bin/ls", ["ls", "/tmp"], ...) = 0
    [pid 1235] openat(AT_FDCWD, "/tmp", O_RDONLY|O_DIRECTORY) = 3
    [pid 1235] getdents64(3, ...) = 1024
    [pid 1234] wait4(1235, ...) = 1235

Pattern:

1. Parent clones child (pid 1235)
2. Child execs `/bin/ls` (replaces process image)
3. Child runs `ls` syscalls
4. Parent waits for child to complete

## Multiple Child Processes

Trace programs that spawn multiple children (e.g., parallel builds):

    $ renacer -f -- make -j4

Tested by: `test_follow_multiple_forks`

Output shows:

    [pid 1234] clone(...) = 1235  # Spawn compiler 1
    [pid 1234] clone(...) = 1236  # Spawn compiler 2
    [pid 1234] clone(...) = 1237  # Spawn compiler 3
    [pid 1234] clone(...) = 1238  # Spawn compiler 4
    [pid 1235] execve("/usr/bin/gcc", ["gcc", "file1.c", ...]) = 0
    [pid 1236] execve("/usr/bin/gcc", ["gcc", "file2.c", ...]) = 0
    [pid 1237] execve("/usr/bin/gcc", ["gcc", "file3.c", ...]) = 0
    [pid 1238] execve("/usr/bin/gcc", ["gcc", "file4.c", ...]) = 0
    # All 4 compilers run in parallel

Use case: Understand parallel build behavior, identify bottlenecks.

## Integration with Other Features

### With Filtering (-e)

Filter syscalls across entire process tree:

    renacer -f -e trace=file -- make test

Tested by: `test_follow_forks_with_filtering`

Output:

- Traces only file operations (open, read, write, close)
- Applies filter to parent + all children
- Useful for debugging I/O issues in multi-process apps

### With Statistics (-c)

Aggregate syscall statistics across all processes:

    renacer -f -c -- make -j8

Tested by: `test_follow_forks_with_statistics`

Example Output:

    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     35.23    0.123456         123      1000         0 read
     28.45    0.099876          99      1005         0 write
     18.32    0.064234          64      1003         0 open
     ...

Statistics include:

- Combined call counts from parent + children
- Total time across all processes
- Unified error counts

Use case: Understand overall resource usage of parallel operations.

### With JSON Output

Export multi-process traces to JSON:

    renacer -f --format json -- ./parallel-app > trace.json

Tested by: `test_follow_forks_with_json`

JSON Structure:

    {
      "pid": 1234,
      "syscall": "clone",
      "result": 1235,
      ...
    },
    {
      "pid": 1235,
      "syscall": "write",
      "arguments": "1, \"child\", 5",
      "result": 5
    }

Key field: `"pid"` distinguishes parent vs child syscalls.

Use case: Programmatic analysis of multi-process behavior.

### With CSV Output

Export for spreadsheet analysis:

    renacer -f --format csv -- make all > build-trace.csv

Tested by: `test_follow_forks_with_csv`

CSV includes PID column:

    pid,syscall,arguments,result
    1234,clone,"CLONE_CHILD_CLEARTID|...",1235
    1235,execve,"/usr/bin/gcc, ...",0
    1235,open,"/tmp/file.c",3

Use case: Analyze build parallelism in Excel/R/Python pandas.

## Edge Cases & Race Conditions

### Immediate Child Exit

Children that exit immediately after fork:

    $ renacer -f -- ./quick-exit-child

Tested by: `test_follow_forks_with_immediate_exit`

Renacer handles race conditions where child exits before tracer attaches:

- Best-effort tracing (may miss some syscalls from very fast children)
- Always traces at least fork/clone event
- Parent trace remains complete

### vfork() Support

vfork() is a lightweight fork variant (shares memory until exec):

    $ renacer -f -- ./vfork-example

Tested by: `test_follow_vfork`

Behavior:

- vfork() appears as `clone(CLONE_VM|CLONE_VFORK|...)`
- Parent suspended until child execs or exits
- Tracer correctly handles suspended parent

### clone() Syscall

On Linux, fork() is implemented via clone():

    $ renacer -f -- ./thread-example

Tested by: `test_follow_clone`

clone() flags reveal process creation type:

- `CLONE_CHILD_CLEARTID|SIGCHLD` - Traditional fork
- `CLONE_VM|CLONE_FS|CLONE_FILES` - Thread creation
- `CLONE_NEWNS|CLONE_NEWPID` - Container/namespace creation

Note: Renacer traces processes, not threads. Thread creation (clone with `CLONE_VM`) may behave differently.

## Practical Examples

### Example 1: Parallel Build Analysis

    $ renacer -f -c -e trace=file -- make -j8

Use case: Understand file I/O patterns in parallel builds.

Output reveals:

- Which files each compiler reads
- File conflicts (multiple processes accessing same file)
- I/O bottlenecks in build system

Example findings:

    [pid 1235] open("/usr/include/stdio.h", O_RDONLY) = 3  # Compiler 1
    [pid 1236] open("/usr/include/stdio.h", O_RDONLY) = 3  # Compiler 2
    [pid 1237] open("/usr/include/stdio.h", O_RDONLY) = 3  # Compiler 3
    # Repeated header reads (ccache could help!)

### Example 2: Shell Script Debugging

    $ renacer -f -- bash ./deploy.sh

Use case: Trace all commands executed by shell script.

Output shows:

    [pid 1234] clone(...) = 1235  # bash forks
    [pid 1235] execve("/usr/bin/rsync", [...]) = 0  # rsync command
    [pid 1235] connect(3, {sa_family=AF_INET, ...}) = 0  # Network call
    [pid 1234] clone(...) = 1236  # bash forks again
    [pid 1236] execve("/usr/bin/ssh", [...]) = 0  # ssh command

Reveals:

- Exact sequence of external commands
- Network operations (rsync, ssh)
- Resource usage per command

### Example 3: Test Suite Profiling

    $ renacer -f -c -T -- cargo test

Use case: Profile test suite parallelism and timing.

Combines:

- `-f`: Trace all test processes (cargo spawns multiple)
- `-c`: Aggregate statistics
- `-T`: Timing data

Output identifies:

- Slowest test processes
- Syscall bottlenecks across tests
- Parallel vs sequential execution patterns

### Example 4: Container Process Tracking

    $ renacer -f -- docker run alpine ls

Use case: Trace container creation and execution.

Output reveals:

    [pid 1234] clone(CLONE_NEWNS|CLONE_NEWPID|...) = 1235  # Container init
    [pid 1235] mount("proc", "/proc", "proc", ...) = 0  # Namespace setup
    [pid 1235] execve("/bin/ls", ["ls"], ...) = 0  # Container command

Shows:

- Container namespace creation (CLONE_NEWNS, CLONE_NEWPID)
- Filesystem mounts
- Actual container command execution

## Troubleshooting

### "Permission denied" Errors

Problem:

    $ renacer -f -- make
    ptrace: Operation not permitted

Causes:

1. Kernel security (Yama ptrace_scope):

       # Check current setting
       cat /proc/sys/kernel/yama/ptrace_scope
       # 0 = unrestricted, 1 = restricted (default on many distros)

   Solution:

       # Temporarily allow ptrace (requires root)
       echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

       # Or run renacer as root (not recommended)
       sudo renacer -f -- make

2. SELinux/AppArmor restrictions: Check security policies.

Tested in: Sprint 18 tests (assume ptrace allowed)

### Missing Child Syscalls

Problem: Child process syscalls not appearing in output.

Possible causes:

1. Child exits very quickly - Race condition (see `test_follow_forks_with_immediate_exit`)

   - Solution: Accept that ultra-fast children may be partially traced

2. Forgot `-f` flag:

       # Wrong (no -f)
       renacer -- make -j8

       # Correct
       renacer -f -- make -j8

3. Thread instead of process:

   - Renacer traces processes (fork/clone with SIGCHLD)
   - Threads (clone with CLONE_VM) may not be fully traced

### Large Output Volume

Problem: Multi-process tracing produces huge output.

Solutions:

1. Filter syscalls:

       renacer -f -e trace=file -- make -j8 > build-io.txt

2. Use statistics mode:

       renacer -f -c -- make -j8
       # Statistics summary instead of full trace

3. Export to structured format:

       renacer -f --format json -- make -j8 | gzip > trace.json.gz

4. Redirect to file:

       renacer -f -- make -j8 > trace.txt 2>&1

### Process Tree Too Deep

Problem: Recursive process creation (fork bombs).

Renacer behavior:

- Traces all descendants (unlimited depth)
- May consume significant system resources

Solution: Use external tools to limit process tree:

    # Limit process tree depth with ulimit
    ulimit -u 100  # Max 100 processes
    renacer -f -- ./potentially-recursive-app

## How It Works

### ptrace Event Tracking

When `-f` is enabled, Renacer:

1. Sets PTRACE_O_TRACEFORK option:

       ptrace(PTRACE_SETOPTIONS, pid, 0,
              PTRACE_O_TRACESYSGOOD |
              PTRACE_O_TRACEFORK |     // Follow fork()
              PTRACE_O_TRACEVFORK |    // Follow vfork()
              PTRACE_O_TRACECLONE |    // Follow clone()
              PTRACE_O_TRACEEXEC);     // Track exec()

2. Receives fork events:

   - Parent makes clone() syscall
   - Kernel sends `PTRACE_EVENT_FORK` to tracer
   - Tracer retrieves child PID

3. Attaches to child:

   - Child automatically stopped by kernel
   - Tracer adds child PID to tracked processes
   - Child resumed and traced independently

4. Parallel tracing:

   - Parent and children traced simultaneously
   - Each process has independent syscall stream
   - PID distinguishes syscalls in output

Implementation: Sprint 18 added ptrace event handling for fork/vfork/clone tracking.

### Performance Overhead

Multi-process tracing overhead:

- ~5-15% per process (similar to single-process tracing)
- Scales linearly with number of processes
- Minimal overhead from fork tracking itself

Example (8-process build):

    # Without tracing
    time make -j8
    real    0m30.0s

    # With multi-process tracing
    time renacer -f -c -- make -j8
    real    0m33.5s  # ~12% overhead (acceptable)

## Performance

- Fork tracking overhead: <1% (just event handling)
- Per-process overhead: 5-15% (syscall tracing)
- Scalability: Tested with 100+ concurrent processes
- Memory: O(N) where N = number of traced processes

Zero overhead when disabled (default behavior without `-f`).

## Summary

Multi-process tracing provides:

- ✅ Automatic fork following with `-f` flag
- ✅ Fork, vfork, clone support (all process creation methods)
- ✅ Fork + exec tracking (child process replacement)
- ✅ Multiple child processes (parallel builds, test suites)
- ✅ Integration with filtering, statistics, JSON/CSV export
- ✅ Race condition handling (immediate child exit)
- ✅ PID tracking in output (distinguish parent vs children)
- ✅ Backward compatible (disabled by default)

All examples tested in: `tests/sprint18_multiprocess_tests.rs` (11+ integration tests)

## Related

- Statistics Mode - Aggregate multi-process stats
- Filtering Syscalls - Filter across process tree
- JSON Output - Export multi-process traces
- CSV Output - Spreadsheet analysis
- Function Profiling - Per-function multi-process analysis