# Scripting Stages

Deep dive into Kelora's Rhai scripting stages: `--begin`, `--filter`, `--exec`, and `--end`.

## Overview

Kelora provides four scripting stages for transforming log data with Rhai scripts:

```
--begin  →  [Process Events]  →  --end
              ↓
           --filter + --exec (per event, in CLI order)
```

| Stage | Runs | Purpose | Access |
|-------|------|---------|--------|
| `--begin` | Once before processing | Initialize state, load data | `conf` map, file helpers |
| `--filter` | Once per event | Select events to keep/skip | `e` (event), `conf`, `meta` |
| `--exec` | Once per event | Transform events | `e` (event), `conf`, `meta`, tracking |
| `--end` | Once after processing | Summarize, report | `metrics`, `conf` |

**Note:** `--filter` and `--exec` can be specified multiple times and execute in exact CLI order.

## Begin Stage

### Purpose

The `--begin` stage runs **once** before any events are processed. Use it to:

- Initialize lookup tables
- Load reference data from files
- Set up shared configuration
- Prepare the `conf` map for use in other stages

### The `conf` Map

The global `conf` map is **read-write** in `--begin` and **read-only** in later stages.

```bash
kelora -j \
    --begin 'conf.valid_users = ["alice", "bob", "charlie"]' \
    --exec 'e.is_valid = e.user in conf.valid_users' \
    app.log
```

### Available Helpers

Special functions available **only** in `--begin`:

#### `read_lines(path)`

Read file as array of strings (one per line, UTF-8).

```bash
kelora -j \
    --begin 'conf.blocked_ips = read_lines("blocked.txt")' \
    --exec 'if e.ip in conf.blocked_ips { e = () }' \
    app.log
```

#### `read_file(path)`

Read entire file as single string (UTF-8).

```bash
kelora -j \
    --begin 'conf.template = read_file("template.txt")' \
    --end 'print(conf.template.replace("{count}", metrics["total"].to_string()))' \
    app.log
```

**Note:** To parse JSON files, use `read_file(path)` and then call `.parse_json()` on the result, or define data structures inline using Rhai's map syntax `#{ key: value }`.

### Examples

#### Load Lookup Table

```bash
kelora -j \
    --begin 'conf.services = #{api: "API Gateway", db: "Database", cache: "Redis"}' \
    --exec 'e.service_name = conf.services.get(e.service, e.service)' \
    app.log
```

#### Load IP Geolocation Data

```bash
kelora -j \
    --begin 'conf.ip_to_country = #{
                "192.168.1.1": "US",
                "10.0.0.1": "UK",
                "172.16.0.1": "DE"
            }' \
    --exec 'e.country = conf.ip_to_country.get(e.ip, "unknown")' \
    app.log
```

#### Initialize Counters

```bash
kelora -j \
    --begin 'conf.start_time = now()' \
    --end 'let duration = now() - conf.start_time; print("Processed in " + duration + "s")' \
    app.log
```

#### Load Configuration

```bash
kelora -j \
    --begin 'conf.threshold = 1000; conf.alert_email = "ops@company.com"' \
    --exec 'if e.duration_ms > conf.threshold { eprint("⚠️  Slow request: " + e.path) }' \
    app.log
```

## Filter Stage

### Purpose

The `--filter` stage runs **once per event** to decide whether to keep or skip it. Use it to:

- Select events matching specific criteria
- Remove unwanted events (debug logs, health checks, etc.)
- Combine multiple filter conditions in sequence
- Control which events reach later `--exec` stages

### Boolean Expressions

Filters must return `true` (keep event) or `false` (skip event):

```bash
kelora -j \
    --filter 'e.level == "ERROR"' \
    app.log
```

**Behavior:**
- Returns `true` → Event passes to next stage
- Returns `false` → Event is skipped (removed from pipeline)
- Error in resilient mode → Treated as `false`, event skipped
- Error in strict mode → Processing aborts

### Access to Event Data

Filters have access to:

- `e` - The current event (read-only)
- `conf` - Configuration from `--begin` (read-only)
- `meta` - Event metadata (line, line_num, filename)

```bash
kelora -j \
    --begin 'conf.min_duration = 1000' \
    --filter 'e.duration_ms > conf.min_duration' \
    app.log
```

### Multiple Filters

Multiple `--filter` flags create an **AND** condition - events must pass all filters:

```bash
kelora -j \
    --filter 'e.level == "ERROR"' \
    --filter 'e.service == "api"' \
    --filter 'e.duration_ms > 1000' \
    app.log
```

Only events that are ERROR **AND** from api service **AND** slow will pass.

### Common Patterns

#### Basic Field Matching

```bash
kelora -j --filter 'e.status >= 400' app.log
kelora -j --filter 'e.user == "admin"' app.log
kelora -j --filter 'e.service in ["api", "db"]' app.log
```

#### String Operations

```bash
kelora -j --filter 'e.message.contains("timeout")' app.log
kelora -j --filter 'e.path.starts_with("/api/")' app.log
kelora -j --filter 'e.level.to_upper() == "ERROR"' app.log
```

#### Regex Matching

```bash
kelora -j --filter 'e.message.has_matches(r"\d{3}-\d{3}-\d{4}")' app.log
kelora -j --filter 'e.ip.has_matches(r"^192\.168\.")' app.log
```

#### Existence Checks

```bash
kelora -j --filter 'e.contains("error")' app.log       # Has 'error' field
kelora -j --filter '"error" in e' app.log              # Same as above
kelora -j --filter 'e.contains("user_id")' app.log     # Has 'user_id' field
```

#### Complex Conditions

```bash
kelora -j \
    --filter '(e.status >= 500) || (e.status >= 400 && e.duration_ms > 5000)' \
    app.log
```

#### Using conf for Dynamic Filtering

```bash
kelora -j \
    --begin 'conf.blocked_ips = ["192.168.1.100", "10.0.0.50"]' \
    --filter '!(e.ip in conf.blocked_ips)' \
    app.log
```

#### Filtering with Metadata

```bash
# Only process events from specific file
kelora -j *.log --filter 'meta.filename == "production.log"'

# Skip first 100 lines
kelora -j app.log --filter 'meta.line_num > 100'
```

### Filter vs --levels

For simple level filtering, `--levels` is more efficient than `--filter`:

**Prefer:**
```bash
kelora -j app.log --levels error,warn
```

**Over:**
```bash
kelora -j app.log --filter 'e.level in ["ERROR", "WARN"]'
```

However, `--filter` provides more flexibility for complex conditions.

Level flags behave like script stages: you can place `--levels` or `--exclude-levels` anywhere in the CLI sequence, and Kelora runs them immediately at that point. Repeat the flag when you want different level rules before and after a transformation (for example, derive a level in `--exec`, then add another `--levels` to act on the new field).

### Filter Output

Filters don't modify events - they only decide pass/skip:

```bash
# This works - filter just checks level
kelora -j --filter 'e.level == "ERROR"' app.log

# This does nothing - assignment in filter has no effect
kelora -j --filter 'e.level = "ERROR"' app.log  # Wrong!
```

To modify events, use `--exec` instead.

## Exec Stage

### Purpose

The `--exec` stage runs **once per event**. Use it to:

- Transform event fields
- Add computed fields
- Filter events (via `e = ()`)
- Track metrics
- Emit multiple events from arrays

### The Event Variable

The current event is available as `e`. Modifications to `e` persist through subsequent `--exec` scripts.

```bash
kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'e.is_slow = e.duration_s > 1.0' \
    app.log
```

### Multiple Exec Scripts

Multiple `--exec` scripts run in order. Each sees changes from previous scripts.

```bash
kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'track_avg("duration", e.duration_s)' \
    --exec 'if e.duration_s > 5.0 { e.alert = true }' \
    app.log
```

**Execution order:**
1. Convert `duration_ms` to `duration_s`
2. Track average duration
3. Add `alert` field for slow requests

### Variable Scope Between Exec Stages

**Important:** Each `--exec` stage has its own isolated scope. Local variables declared with `let` do **NOT** persist between stages:

```bash
# ❌ WRONG - ctx doesn't exist in the second stage
kelora -j \
    --exec 'let ctx = e.user_id' \
    --exec 'e.context = ctx' \  # ERROR: ctx undefined!
    app.log
```

**What persists between stages:**
- ✅ **Event fields** (`e.field = value`) - Modifications carry forward
- ✅ **`conf` map** - Initialized in `--begin`, read-only in exec stages
- ✅ **`metrics` map** - Populated by `track_*()` functions
- ✅ **`window` array** - When using `--window`

**What does NOT persist:**
- ❌ **Local variables** (`let x = ...`) - Scoped to the stage only
- ❌ **Function definitions** - Unless loaded via `--include`

**Solution - Use semicolons for shared variables:**

When you need local variables to persist across operations, use semicolons within a single `--exec`:

```bash
# ✅ CORRECT - Both operations in one stage
kelora -j \
    --exec 'let ctx = e.user_id; e.context = ctx' \
    app.log
```

**When to use multiple stages vs semicolons:**

Use **multiple `-e` stages** when you want:
- Stage-level error isolation (see resilient mode below)
- Logical separation of transformation steps
- Progressive validation checkpoints

Use **semicolons within one `-e`** when you need:
- Shared local variables across operations
- All-or-nothing execution (no partial results)

### Intermixing --filter and --exec

`--filter` and `--exec` are both script stages and execute in **exact CLI order**:

```bash
kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \    # Stage 1: Transform all events
    --filter 'e.duration_s > 1.0' \                    # Stage 2: Keep only slow events
    --exec 'track_count("slow")' \                     # Stage 3: Track (slow only)
    --exec 'e.alert = true' \                          # Stage 4: Add field (slow only)
    app.log
```

**What happens:**
1. First `--exec` adds `duration_s` field to all events
2. `--filter` removes events under 1.0s
3. Second `--exec` only processes slow events (tracks count)
4. Third `--exec` only processes slow events (adds alert field)

Later stages only see events that passed earlier filters. This allows precise control over which events are transformed or tracked.

### Atomic Execution

In resilient mode (default), exec scripts execute **atomically**:

- If an error occurs, changes are rolled back
- Original event is returned unchanged
- Processing continues with next event

```bash
kelora -j \
    --exec 'e.result = e.value.to_int() * 2' \
    app.log
```

If `e.value` is not a valid integer:

- Error is recorded
- Event passes through unchanged
- No partial modifications

In strict mode (`--strict`), errors abort immediately.

### Resilient Mode and Stage Snapshotting

Resilient mode creates **snapshots after each successful stage**. If a later stage fails, the event **reverts to the last successful snapshot**:

```bash
kelora -j --resilient \
    --exec 'e.step1 = "processed"' \        # Snapshot 1
    --exec 'e.step2 = risky_parse(e.raw)' \ # Might fail
    --exec 'e.step3 = "complete"' \         # Snapshot 3 (if step2 succeeds)
    app.log
```

**Behavior on error:**
- If `step2` fails for an event, it **keeps `step1` but not `step2`**
- The event continues through the pipeline with fields from the last successful stage
- Later stages see the rolled-back event

**Why use multiple stages for error handling:**

```bash
# Multiple stages - Graceful degradation
kelora -j --resilient \
    --exec 'e.safe = "always_set"' \        # Always succeeds
    --exec 'e.parsed = parse_json(e.raw)' \ # Might fail for some events
    --exec 'track_count(e.parsed.type)' \   # Only runs if parsing succeeded
    -k safe,parsed
# Events that fail parsing still have 'safe' field and appear in output

# Single stage - All-or-nothing
kelora -j --resilient \
    --exec 'e.safe = "always_set"; e.parsed = parse_json(e.raw); track_count(e.parsed.type)' \
    -k safe,parsed
# If parsing fails, the ENTIRE exec fails - no 'safe' field, event unchanged
```

**Design pattern - Progressive risk:**

Place risky operations in later stages so earlier transformations survive:

```bash
kelora -j --resilient \
    --exec 'e.normalized = e.status.to_string()' \  # Safe normalization
    --exec 'e.enriched = lookup_external(e.id)' \   # Risky external call
    --exec 'e.computed = e.enriched.calculate()' \  # Depends on risky data
    app.log
```

Events that fail external lookup still get `normalized` field.

**Trade-off:**
- **Multiple stages**: Better error isolation, partial success, but local variables don't persist
- **Single stage with semicolons**: Shared variables, but all-or-nothing execution

### Common Patterns

#### Transform Fields

```bash
kelora -j \
    --exec 'e.level = e.level.to_upper()' \
    --exec 'e.message = e.message.trim()' \
    app.log
```

#### Add Computed Fields

```bash
kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'e.timestamp_unix = e.timestamp.to_unix()' \
    app.log
```

#### Conditional Field Creation

```bash
kelora -j \
    --exec 'if e.status >= 500 { e.severity = "critical" } else if e.status >= 400 { e.severity = "error" }' \
    app.log
```

#### Remove Events

```bash
kelora -j \
    --exec 'if e.level == "DEBUG" { e = () }' \
    app.log
```

#### Track Metrics

```bash
kelora -j \
    --exec 'track_count(e.service)' \
    --exec 'track_avg("response_time", e.duration_ms)' \
    --metrics \
    app.log
```

#### Fan-Out Arrays

```bash
kelora -j \
    --exec 'emit_each(e.items)' \
    app.log
```

Each array element becomes a separate event.

### Access to conf

The `conf` map from `--begin` is **read-only** in `--exec`:

```bash
kelora -j \
    --begin 'conf.multiplier = 2.5' \
    --exec 'e.adjusted = e.value * conf.multiplier' \
    app.log
```

### Access to meta

The `meta` variable provides event metadata in `--exec` and `--filter`:

```bash
kelora -j \
    --exec 'e.source = meta.filename' \
    server1.log server2.log
```

**Available metadata attributes:**

- `meta.line` - Original raw line from input (always available)
- `meta.line_num` - Line number, 1-based (available when processing files)
- `meta.filename` - Source filename (available with multiple files or explicit file arguments)

**Multi-file tracking example:**

```bash
kelora -j logs/*.log --metrics \
    --exec 'if e.level == "ERROR" { track_count(meta.filename) }' \
    --end 'for file in metrics.keys() { print(file + ": " + metrics[file] + " errors") }'
```

**Debugging with line numbers:**

```bash
kelora -j --filter 'e.status >= 500' \
    --exec 'eprint("⚠️  Server error at " + meta.filename + ":" + meta.line_num)' \
    app.log
```

**Re-parsing with raw line:**

```bash
kelora -j \
    --exec 'if e.message.contains("CUSTOM:") { e.custom = meta.line.after("CUSTOM:").parse_json() }' \
    app.log
```

## End Stage

### Purpose

The `--end` stage runs **once** after all events are processed. Use it to:

- Summarize metrics
- Generate reports
- Print final statistics
- Export aggregated data

### The metrics Map

The global `metrics` map contains all tracked data from `track_*()` functions:

```bash
kelora -j \
    --exec 'track_count(e.service)' \
    --end 'for key in metrics.keys() { print(key + ": " + metrics[key]) }' \
    app.log
```

### Available Data

In `--end`, you have access to:

- `metrics` - All tracked metrics (counts, sums, averages, etc.)
- `conf` - Read-only configuration from `--begin`
- Standard Rhai functions (print, file helpers if `--allow-fs-writes`)

### Examples

#### Print Summary Statistics

```bash
kelora -j \
    --exec 'track_count("total"); if e.level == "ERROR" { track_count("errors") }' \
    --end 'let error_rate = metrics.errors / metrics.total * 100; print("Error rate: " + error_rate + "%")' \
    app.log
```

#### Generate Report

```bash
kelora -j \
    --exec 'track_count(e.service)' \
    --end 'print("=== Service Report ==="); for svc in metrics.keys() { print(svc + ": " + metrics[svc] + " requests") }' \
    app.log
```

#### Export Metrics to File

```bash
kelora -j --allow-fs-writes \
    --exec 'track_count(e.service)' \
    --end 'append_file("report.txt", "Total services: " + metrics.len().to_string())' \
    app.log
```

#### Calculate Percentages

```bash
kelora -j \
    --exec 'track_count("total"); track_count(e.level)' \
    --end 'for level in ["INFO", "WARN", "ERROR"] { let pct = metrics.get(level, 0) / metrics.total * 100; print(level + ": " + pct + "%") }' \
    app.log
```

## Stage Interaction

### Data Flow Between Stages

```
--begin:  Initialize conf map
    ↓
    conf (read-only)
    ↓
--exec:   Process events, track metrics
    ↓
    metrics + conf (both read-only)
    ↓
--end:    Summarize and report
```

### Complete Example

```bash
kelora -j \
    --begin 'conf.threshold = 1000; conf.start = now()' \
    --exec 'if e.duration_ms > conf.threshold { track_count("slow") }' \
    --exec 'track_count("total")' \
    --end 'let elapsed = now() - conf.start; print("Processed " + metrics.total + " events in " + elapsed + "s"); print("Slow requests: " + metrics.get("slow", 0))' \
    app.log
```

**Flow:**
1. `--begin`: Set threshold to 1000ms, record start time
2. `--exec` (per event): Track slow requests, track total
3. `--end`: Calculate elapsed time, print summary

## Using Exec Files

### `-E, --exec-file`

Load Rhai script from file for the exec stage:

**transform.rhai:**
```rhai
// Convert duration to seconds
e.duration_s = e.duration_ms / 1000;

// Add severity based on status
if e.status >= 500 {
    e.severity = "critical";
} else if e.status >= 400 {
    e.severity = "error";
} else {
    e.severity = "ok";
}

// Track metrics
track_count(e.severity);
track_avg("response_time", e.duration_s);
```

**Usage:**
```bash
kelora -j -E transform.rhai --metrics app.log
```

### `-I, --include`

Include Rhai library files before script stages:

**helpers.rhai:**
```rhai
fn classify_status(status) {
    if status >= 500 {
        "server_error"
    } else if status >= 400 {
        "client_error"
    } else if status >= 300 {
        "redirect"
    } else if status >= 200 {
        "success"
    } else {
        "other"
    }
}
```

**Usage:**
```bash
kelora -j \
    -I helpers.rhai \
    --exec 'e.status_class = classify_status(e.status)' \
    app.log
```

## Best Practices

### Use --begin for Initialization

**Good:**
```bash
kelora -j \
    --begin 'conf.lookup = read_json("data.json")' \
    --exec 'e.name = conf.lookup.get(e.id, "unknown")' \
    app.log
```

**Bad:**
```bash
kelora -j \
    --exec 'let lookup = read_json("data.json"); e.name = lookup.get(e.id, "unknown")' \
    app.log
```

The bad example reads the file **once per event** (slow and wasteful).

### Keep --exec Scripts Simple

Break complex logic into multiple `--exec` scripts:

**Good:**
```bash
kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'e.is_slow = e.duration_s > 1.0' \
    --exec 'if e.is_slow { track_count("slow_requests") }' \
    app.log
```

**Bad:**
```bash
kelora -j \
    --exec 'e.duration_s = e.duration_ms / 1000; e.is_slow = e.duration_s > 1.0; if e.is_slow { track_count("slow_requests") }' \
    app.log
```

The good example is easier to read and debug.

### Use --end for Summaries

**Good:**
```bash
kelora -j \
    --exec 'track_count(e.service)' \
    --end 'print("Total services: " + metrics.len())' \
    app.log
```

**Bad:**
```bash
kelora -j \
    --exec 'track_count(e.service); print("Processing...")' \
    app.log
```

The bad example prints on every event (noisy).

### Leverage File Helpers

For complex logic, use `-E` and `-I`:

```bash
kelora -j -I helpers.rhai -E transform.rhai --metrics app.log
```

This keeps command lines clean and logic maintainable.

## Performance Considerations

### Begin Stage Overhead

The `--begin` stage runs once, so file I/O here is acceptable:

```bash
kelora -j \
    --begin 'conf.large_dataset = read_json("10mb.json")' \
    --exec 'e.enriched = conf.large_dataset.get(e.id, #{})' \
    app.log
```

### Exec Stage Optimization

The `--exec` stage runs per event. Avoid expensive operations:

**Slow:**
```bash
kelora -j \
    --exec 'let lookup = read_json("data.json"); e.name = lookup.get(e.id, "unknown")' \
    app.log
```

**Fast:**
```bash
kelora -j \
    --begin 'conf.lookup = read_json("data.json")' \
    --exec 'e.name = conf.lookup.get(e.id, "unknown")' \
    app.log
```

### End Stage Overhead

The `--end` stage runs once, so complex calculations are fine:

```bash
kelora -j \
    --exec 'track_count(e.service)' \
    --end 'let sorted = metrics.keys().sort(); for key in sorted { print(key + ": " + metrics[key]) }' \
    app.log
```

## Parallel Processing

When using `--parallel`, scripting stages behave differently:

### Begin and End

`--begin` and `--end` run **once** (not parallelized):

```bash
kelora -j --parallel \
    --begin 'conf.start = now()' \
    --exec 'track_count(e.service)' \
    --end 'print("Duration: " + (now() - conf.start))' \
    app.log
```

### Exec Stage

`--exec` runs in parallel across worker threads:

- Each thread has its own copy of `conf` (read-only)
- Tracking functions aggregate across threads
- Event modifications are isolated per thread

```bash
kelora -j --parallel \
    --exec 'e.duration_s = e.duration_ms / 1000' \
    --exec 'track_count(e.service)' \
    app.log
```

### Thread Safety

Kelora handles thread safety automatically:

- `conf` is cloned per thread (immutable)
- `metrics` uses thread-safe aggregation
- Event modifications are isolated

You don't need to worry about race conditions in scripts.

## Troubleshooting

### conf is Read-Only in --exec

**Problem:**
```bash
kelora -j --exec 'conf.value = 42' app.log
# Error: conf is read-only in exec stage
```

**Solution:** Initialize in `--begin`:
```bash
kelora -j --begin 'conf.value = 42' --exec 'e.result = conf.value * 2' app.log
```

### metrics Not Available in --exec

**Problem:**
```bash
kelora -j --exec 'print(metrics["total"])' app.log
# Error: metrics not available in exec stage
```

**Solution:** Use `--end`:
```bash
kelora -j --exec 'track_count("total")' --end 'print(metrics["total"])' app.log
```

### File Helpers Not Working

**Problem:**
```bash
kelora -j --exec 'append_file("out.txt", e.message)' app.log
# Error: filesystem writes not allowed
```

**Solution:** Add `--allow-fs-writes`:
```bash
kelora -j --allow-fs-writes --exec 'append_file("out.txt", e.message)' app.log
```

### Script Stage Ordering

**Understanding:** `--filter` and `--exec` execute in **exact CLI order**, intermixed.

```bash
# Both --exec scripts run on all events
kelora -j --exec 'e.a = 1' --exec 'e.b = e.a + 1' app.log

# Filter runs between execs - second --exec only sees filtered events
kelora -j --exec 'e.a = 1' --filter 'e.level == "ERROR"' --exec 'e.b = e.a + 1' app.log
```

**What happens in the second example:**
1. First `--exec` adds field `a=1` to all events
2. `--filter` removes non-ERROR events
3. Second `--exec` adds field `b=2` to ERROR events only

This is **expected behavior** - later stages only process events that passed earlier filters. Each stage (filter or exec) processes the output of the previous stage sequentially.

## See Also

- [Processing Architecture](pipeline-model.md) - Three-layer processing model and script stage ordering
- [Events and Fields](events-and-fields.md) - Working with event data
- [Function Reference](../reference/functions.md) - All available Rhai functions
- [CLI Reference](../reference/cli-reference.md) - Complete flag documentation
