# 07: Process and Pipeline

## Overview

`decree process` batch-processes all unprocessed migrations and then
drains the inbox of any spawned messages. This is the primary command
for executing work — no daemon required for basic use. The core message
processing loop handles lifecycle hooks (migration 10), routine
execution (migration 04), retry strategy, and dead-lettering.

Decree's core has zero awareness of git or any VCS. All state management
(snapshots, reverts, checkpoints) is delegated entirely to lifecycle
hooks. The core only orchestrates: run hooks, run routines, handle
retries, manage the inbox.

## `decree process` — Batch Processing + Inbox Drain

```
decree process
decree              # bare `decree` is equivalent
```

`decree process`:
1. Run `beforeAll` hook (if configured)
2. Read `.decree/processed.md` to find unprocessed migrations
3. For the first unprocessed migration (alphabetically):
   - Generate a new chain ID
   - Create an inbox message (`seq: 0`) with the full migration
     content copied into the message body
   - Process the message and any follow-up messages in its chain
   - On success: mark migration as processed in `.decree/processed.md`
4. **Drain the inbox**: process all remaining messages in `.decree/inbox/`
   before moving to the next migration. Messages are processed LIFO
   (last in, first out) — newest messages are processed first. Within
   the same chain, messages are processed depth-first.
5. Move to the next unprocessed migration (new chain, new chain ID)
6. Repeat steps 3–5 until all migrations are processed and inbox is empty
7. Run `afterAll` hook (if configured)
8. Print total duration summary:
   `Processed 4 migrations in 23m15s`

Each migration starts a new chain. If a migration fails all retries,
its message is dead-lettered and processing continues (drain inbox,
then next migration).

> **IMPORTANT**: Migrations are added to the inbox ONE AT A TIME. Each
> migration's chain must fully complete (including all outbox → inbox
> follow-ups) and the inbox must be completely empty before the next
> migration is added.

### Migration → Inbox Message

When decree creates an inbox message from a migration file:
- `id`, `chain`, `seq`: generated by decree
- `routine`: from migration frontmatter (or resolved later)
- `migration`: set to the original migration filename
- **Custom frontmatter fields**: all non-standard fields from the
  migration's YAML frontmatter are copied into the inbox message and
  subsequently passed as env vars to the routine. For example, a
  migration with `project_name: myapp` results in
  `project_name=myapp` in the routine's environment.

This is essential — migration authors need custom fields to
parameterize routines without modifying routine scripts.

### Migration Tracking

When a migration's root message (and its entire chain) completes
successfully, decree appends the migration's original filename
(e.g. `01-add-auth.md`) to `.decree/processed.md`. The inbox message
carries the migration filename in a `migration` frontmatter field set
by decree when creating the message from the migration file.

## Message Processing

The core processing logic shared by `decree process` and `decree daemon`:

```
take next message from inbox
  (LIFO ordering; prefer messages from current chain — depth-first)
normalize message (fill missing fields — see migration 03)
create .decree/runs/<chain>-<seq>/
copy normalized message -> .decree/runs/<chain>-<seq>/message.md

1. Run beforeEach hook (if configured)
     Env vars include: DECREE_ATTEMPT=1
2. Determine routine (from normalized message frontmatter)
3. Resolve routine file (see migration 04)
4. Execute routine:
   - Write start timestamp to log: `[decree] start 2026-03-03T14:32:00`
   - bash .decree/routines/<routine>.sh 2>&1 | tee <msg-dir>/<log-file>
   - Write end timing to log:
     `[decree] duration 4m32s end 2026-03-03T14:36:32`
   - Log file: routine.log (attempt 1), routine-2.log (attempt 2), etc.
   - Env vars: message_file, message_id, message_dir,
     chain, seq, plus custom variables
   - Working directory: project root
   - Output is streamed to the terminal in real-time AND logged
   - After execution: truncate log if it exceeds max_log_size
   - The routine runs as a child process in the same process group
   - Ctrl-C (SIGINT) kills the routine, its subprocesses, and exits decree immediately

5. On SUCCESS:
   - Run afterEach hook (DECREE_ROUTINE_EXIT_CODE=0)
   - Collect outbox (see below)
   - Delete message from .decree/inbox/ (run directory is the permanent record)
   - If message has `migration` field: append value to .decree/processed.md
6. On FAILURE (see "Retry Strategy")
7. On EXHAUSTION (all retries failed):
   - Run afterEach hook (DECREE_ROUTINE_EXIT_CODE=<code>,
     DECREE_ATTEMPT=max_retries)
   - Clear outbox (discard follow-ups from failed routine)
   - Move message to .decree/inbox/dead/
   - Continue to next message
8. Check inbox for new messages in this chain
   - If found: process depth-first
```

## Retry Strategy

When a routine fails (non-zero exit code):

1. Run afterEach hook with `DECREE_ROUTINE_EXIT_CODE=<exit_code>` and
   `DECREE_ATTEMPT=<current>`
2. If attempts remain, increment attempt counter
3. Run beforeEach hook with `DECREE_ATTEMPT=<next>`
4. Re-execute the routine

Hooks receive enough context via env vars to implement their own
state management strategy (e.g. git stash hooks can decide whether
to revert or keep changes based on attempt number).

**Per-attempt log files**: Each retry attempt gets its own log file.
The first attempt writes to `routine.log`, subsequent attempts write
to `routine-2.log`, `routine-3.log`, etc.

## Hook Environment Variables

Decree passes these env vars to all lifecycle hooks (in addition to
standard message parameters):

| Variable | Description |
|---|---|
| `DECREE_ATTEMPT` | Current attempt number (1-indexed) |
| `DECREE_MAX_RETRIES` | Configured max retries from config |
| `DECREE_ROUTINE_EXIT_CODE` | Exit code of the routine (afterEach only) |
| `DECREE_HOOK` | Hook type name |

## Dead-Letter Directory

Messages moved to `.decree/inbox/dead/` when:
- All retries exhausted
- `MaxDepthExceeded` triggered
- Normalization fails (e.g. `RoutineNotFound`)

Users can re-queue by moving back to `.decree/inbox/`.

## Outbox Collection

After a routine succeeds, decree collects follow-up messages:

1. Scan `.decree/outbox/` for `*.md` files (sorted alphabetically).
   If any non-`.md` files are found, print a warning:
   `Warning: non-.md file in outbox ignored: <filename>`
2. For each `.md` file, starting at `next_seq = current seq + 1`:
   - Check depth: if `next_seq >= max_depth`, move the outbox file
     to `.decree/outbox/dead/` with a `MaxDepthExceeded` warning
     and skip it
   - `chain`: current chain ID (overrides any frontmatter value)
   - `seq`: `next_seq` (overrides any frontmatter value), then increment
   - `routine`: from outbox file frontmatter if present
   - Custom frontmatter fields are preserved
   - Body: body of the outbox file
3. Write to `.decree/inbox/<chain>-<seq>.md`
   (each file gets a unique seq, so no collisions)
4. Clear `.decree/outbox/` (only successfully collected files)

On exhaustion (all retries failed), the outbox is cleared without
collecting — follow-ups from a failed routine are discarded.

## Depth Limiting

Sequence number acts as depth counter:
- Root messages: `seq: 0`
- Decree increments `seq` by 1 when collecting outbox files
- Routines never set or increment `seq`
- When `next_seq >= max_depth` (default: 10) during outbox collection:
  the file is moved to `.decree/outbox/dead/` instead of the inbox
- This is checked at collection time, not processing time

## Routine Execution and Signal Handling

### Real-Time Output

Routines are executed via `bash <routine>.sh 2>&1 | tee <msg-dir>/routine.log`.
This means:
- The AI tool's streaming output (reasoning, tool calls, progress) is
  visible in the terminal as it happens
- The same output is simultaneously written to `routine.log`
- The user can watch the AI work in real-time and review logs later

### Log Size Management

AI tools can produce enormous output (reasoning traces, file contents,
tool calls). A single AI invocation can easily produce hundreds of KB.
With multiple AI calls per routine and retries, logs can explode to
many MB.

**Per-attempt log files**: Each retry attempt gets its own log file.
The first attempt writes to `routine.log`, subsequent attempts write
to `routine-2.log`, `routine-3.log`, etc. This prevents retry
accumulation and makes it easy to compare attempts.

**Tail-truncation**: After each routine execution completes (success
or failure), decree checks the log file size against `max_log_size`
(default: 2MB). If the log exceeds the limit, truncate from the head
and prepend a marker line:

```
[log truncated — showing last 2MB of output]
```

The tail is kept because the end of execution (errors, final output)
is most useful for debugging. Real-time terminal output is unaffected —
truncation only applies to the saved log file.

**Config**: `max_log_size` in `config.yml` (bytes, default: `2097152`
= 2MB). Set to `0` to disable truncation.

### Ctrl-C / SIGINT

Ctrl-C must immediately end the entire decree session. The user
expects Ctrl-C to stop the AI and return to their shell — not to
silently continue processing the next message or retry.

When SIGINT is received, decree must:
1. Kill the running routine and all its child processes
2. Clean up the current run directory (write end timestamp to log)
3. Exit immediately — do **not** retry, do **not** process more
   messages, do **not** run afterEach/afterAll hooks
4. Exit with code 130 (standard SIGINT exit code: 128 + 2)

This is the single most important UX guarantee: when the user hits
Ctrl-C, everything stops. No background processing continues.

#### Implementation Requirements

The following implementation details are **mandatory** — they are the
only known-correct approach for reliable signal handling. Previous
attempts using simpler approaches all failed.

**1. Child process group isolation.** The routine's bash process must
be spawned in its **own** process group using `process_group(0)` (Rust
stable since 1.64). Do NOT keep the child in decree's process group —
terminal SIGINT delivery to a shared group is unreliable across
platforms and shell configurations. Instead, decree explicitly manages
child lifecycle.

**2. Non-blocking child execution.** Use `spawn()` + `try_wait()` in
a poll loop. Do **NOT** use `cmd.status()` — it blocks the thread,
preventing decree from checking the shutdown flag. The poll loop
should sleep ~10ms between checks to avoid busy-waiting.

```
let mut child = cmd.spawn()?;
let child_id = child.id();
loop {
    match child.try_wait()? {
        Some(status) => break status.code(),
        None => {
            if shutdown_requested {
                kill_child_group(child_id);
                child.wait();  // reap zombie
                return exit_code_130;
            }
            sleep(10ms);
        }
    }
}
```

**3. Signal handler forwards SIGTERM to child group.** Register a
SIGINT handler that (a) sets an atomic shutdown flag AND (b) sends
SIGTERM (not SIGINT) to the child's process group via
`kill(-child_pid, SIGTERM)`. The negative PID targets the entire
process group. Use SIGTERM because bash scripts handle it more
reliably than SIGINT, and `set -e` scripts may not propagate SIGINT
correctly.

**4. Global atomic for child PID.** The signal handler and the
execution loop run in different contexts (signal handler vs main
thread). Use a global `AtomicU32` to share the current child PID.
Store the PID after `spawn()`, clear it (set to 0) after `wait()`.
The signal handler reads this to know which process group to kill.

**Why these requirements exist:**
- `cmd.status()` blocks → decree never checks the shutdown flag →
  SIGINT is silently ignored until the child exits on its own
- Keeping the child in decree's process group → SIGINT may or may not
  reach children depending on terminal/shell behavior → orphaned
  processes
- Sending SIGINT instead of SIGTERM → bash `set -e` scripts may not
  propagate it to subprocesses → orphaned `sleep`, `tee`, AI processes
- Without `process_group(0)` → `kill(-pid, SIGTERM)` kills decree's
  own group instead of just the child tree → self-termination

#### Testing Ctrl-C behavior

Ctrl-C handling must be explicitly verified through automated tests:

- **Unit test**: Send SIGINT to decree while a routine is running.
  Verify decree exits with code 130 and does not process additional
  messages or retries.
- **Subprocess propagation test**: Start a routine that spawns a
  long-running subprocess (e.g. `sleep 3600`). Send SIGINT to decree.
  Verify both the routine and its subprocess are killed (no orphaned
  processes).
- **No-retry test**: Queue two messages. Start processing the first.
  Send SIGINT during the first message's routine. Verify the second
  message is never processed.
- **Clean exit test**: Send SIGINT during routine execution. Verify
  the log file contains the end timestamp and decree exits cleanly
  (no panic, no partial writes to `processed.md`).

## Message Directory Structure

```
.decree/runs/
├── D0001-1432-01-add-auth-0/
│   ├── message.md
│   ├── routine.log             # attempt 1 (includes start/end timestamps)
│   └── routine-2.log           # attempt 2 (if retried)
├── D0001-1432-01-add-auth-1/
│   └── ...
└── D0001-1435-02-add-database-0/
    └── ...
```

Routine template authors should reference `${message_dir}` for context
on prior attempts. Log files from previous attempts are available in
the run directory and can be passed to the AI tool for debugging
context on retries.

## `decree process --dry-run`

Lists unprocessed migrations and resolves which routine each would use,
without creating any messages or writing any files.

1. List unprocessed migrations in order
2. Resolve which routine each would use (frontmatter → config default)
3. Run pre-checks for those specific routines
4. Report results
5. Exit 0 if all pass, 3 if any fail

```
$ decree process --dry-run

Dry run — no messages will be created:
  01-add-auth.md        → develop        PASS
  02-add-database.md    → rust-develop   PASS
  03-add-api.md         → develop        FAIL: opencode not found

Pre-check failures: 1 of 3
```

## Progress Indicator

**TTY**: sticky bottom status line using ANSI scroll margins / cursor
positioning. Format:

```
[Migration 2/4: 03-add-api.md → develop]
```

Updates on: new migration, follow-up, retry.

- Follow-up: `[Migration 2/4: 03-add-api.md → develop (follow-up 1)]`
- Retry: `[Migration 2/4: 03-add-api.md → develop (attempt 2/3)]`

**Non-TTY**: plain status line printed before each migration.

## Acceptance Criteria

- [ ] Migration content is copied into the inbox message body (not just referenced)
- [ ] Inbox message carries `migration` frontmatter field with original filename
- [ ] Custom frontmatter fields from migration files are preserved in the inbox message
- [ ] Custom fields from migrations are passed as env vars to routines
- [ ] `decree process` processes unprocessed migrations in alphabetical order
- [ ] Inbox is fully drained after each migration before the next migration starts
- [ ] Inbox drain uses LIFO ordering (newest first, depth-first within chains)
- [ ] Bare `decree` is equivalent to `decree process`
- [ ] Each migration gets a new chain ID
- [ ] Run directory `.decree/runs/<chain>-<seq>/` is created per message
- [ ] Routine output streams to terminal in real-time via `tee`
- [ ] Routine output is simultaneously logged to `routine.log`
- [ ] Each retry attempt writes to a separate log (`routine.log`, `routine-2.log`, etc.)
- [ ] Logs exceeding `max_log_size` are tail-truncated with a marker line
- [ ] `max_log_size: 0` disables truncation
- [ ] Ctrl-C kills the routine, its subprocesses, and exits decree with code 130
- [ ] Ctrl-C does not trigger retries, further message processing, or afterEach/afterAll hooks
- [ ] Ctrl-C propagates to all child processes (no orphaned subprocesses)
- [ ] Ctrl-C during routine execution writes end timestamp to log before exiting
- [ ] Ctrl-C does not partially write to `processed.md`
- [ ] SIGINT handling is verified by automated tests (unit + integration)
- [ ] Routine child process is spawned in its own process group (`process_group(0)`)
- [ ] Routine execution uses non-blocking `spawn()` + `try_wait()` poll loop (NOT `status()`)
- [ ] Signal handler forwards SIGTERM to child process group via `kill(-pid, SIGTERM)`
- [ ] Global atomic shares child PID between signal handler and execution loop
- [ ] Lifecycle hooks run in correct order around each message
- [ ] Hooks receive `DECREE_ATTEMPT`, `DECREE_MAX_RETRIES`, `DECREE_ROUTINE_EXIT_CODE`
- [ ] Decree core has zero git/VCS awareness — all state management is in hooks
- [ ] Follow-up messages are processed depth-first within chains
- [ ] Exhausted messages are dead-lettered, processing continues
- [ ] `MaxDepthExceeded` at outbox collection moves file to `outbox/dead/`
- [ ] Dead-lettered messages can be re-queued by moving back to inbox
- [ ] Empty message bodies are accepted and processed normally
- [ ] Outbox only collects `*.md` files
- [ ] Non-`.md` files in outbox trigger a warning
- [ ] Successful messages are deleted from inbox (not moved to done/)
- [ ] Run directory is the permanent record of processing
- [ ] Migrations are added to inbox one at a time
- [ ] `decree process --dry-run` lists migrations and resolves routines
- [ ] `decree process --dry-run` runs pre-checks and reports results
- [ ] `decree process --dry-run` exits 0 if all pass, 3 if any fail
- [ ] `decree process --dry-run` creates no messages and writes no files
- [ ] TTY progress indicator shows sticky bottom status line
- [ ] Progress indicator updates on new migration, follow-up, and retry
- [ ] Non-TTY progress prints plain status line before each migration
- [ ] Each routine log starts with `[decree] start <ISO 8601>` timestamp
- [ ] Each routine log ends with `[decree] duration <elapsed> end <ISO 8601>`
- [ ] `decree process` prints total duration summary after all migrations complete
