# WarpDrive Architecture: Immutable Infrastructure Design

## Design Philosophy

WarpDrive is built for **immutable infrastructure** with a **plugin-first architecture**:

1. **Read-Only Filesystem** - Designed to run in Docker with read-only root FS
2. **No Hot Reload** - Configuration changes = new container deployment
3. **Plugin System** - Auth, middleware, custom logic via WASM plugins
4. **Hook System** - Lifecycle events with leader election support

---

## Core Architecture Principles

### 1. Immutable Deployments

```
Config Change Flow:
1. Edit warpdrive.toml
2. Build new Docker image (or ConfigMap update)
3. Deploy new container
4. Old container terminates gracefully
5. Zero trust in runtime file modification
```

**Why No Hot Reload?**
- Ensures config matches what was tested in CI/CD
- Prevents config drift between instances
- Immutable = auditable and reproducible

### 2. Extension via Plugins (Not Built-ins)

**Philosophy:** Keep core minimal, extend via WASM

| Feature | Aralez Approach | WarpDrive Approach |
|---------|----------------|-------------------|
| **Auth** | Built-in (Basic/JWT) | WASM plugin |
| **Rate Limiting** | Built-in per-path | WASM plugin + hooks |
| **Custom Logic** | Fork + rebuild | Load WASM plugin |
| **Middleware** | Core feature set | Plugin system |

---

## Hook System

### Hook Phases

```rust
pub enum HookPhase {
    PreStart,      // Before server starts (migrations, warmup)
    ClusterReady,  // After all instances ready
    PreShutdown,   // Before graceful shutdown
    PostShutdown,  // After shutdown complete
}
```

### Hook Configuration

```toml
[[hooks]]
name = "run_migrations"
phase = "pre_start"
command = "bundle exec rails db:migrate"
working_dir = "/app"
timeout_secs = 300
require_leader = true  # Only one instance executes
continue_on_failure = false

[[hooks]]
name = "clear_cache"
phase = "pre_shutdown"
command = "/scripts/flush_redis.sh"
timeout_secs = 30
require_leader = false  # All instances execute
```

### Leader Election

Hooks with `require_leader = true` use **Redis-based leader election**:

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│ Instance A  │     │ Instance B  │     │ Instance C  │
└─────┬───────┘     └─────┬───────┘     └─────┬───────┘
      │                   │                   │
      ├──────────────────►│◄──────────────────┤
      │   Redis Lock      │                   │
      │   (Leader: A)     │                   │
      │                   │                   │
      ▼                   ▼                   ▼
  ✓ Runs Hook       Waits & Monitors    Waits & Monitors
```

**Environment Variables Injected:**
- `WARPDRIVE_CLUSTER_NAME` - Cluster identifier
- `WARPDRIVE_INSTANCE_ID` - Unique instance ID
- `WARPDRIVE_HOOK_NAME` - Hook name
- `WARPDRIVE_HOOK_PHASE` - Execution phase

### Hook State Machine

```
┌─────────┐
│ Pending │
└────┬────┘
     │
     ├─────────────┬────────────────┐
     │             │                │
require_leader?    │            start
     │             │                │
     ▼             ▼                ▼
┌────────────┐    ...          ┌─────────┐
│ Waiting    │                 │ Running │
│ ForLeader  │                 └────┬────┘
└────┬───────┘                      │
     │                               │
became_leader                  success?
     │                               │
     ▼                       ┌───────┴────────┐
┌─────────┐                 ▼                ▼
│ Running │            ┌───────────┐    ┌────────┐
└────┬────┘            │ Completed │    │ Failed │
     │                 └───────────┘    └────────┘
     ▼
┌───────────┐
│ Completed │
└───────────┘
```

---

## WASM Plugin System

### Plugin Architecture

```
┌────────────────────────────────────────────┐
│           WarpDrive Core                   │
│                                            │
│  ┌──────────────────────────────────────┐ │
│  │      Wasmtime Runtime                │ │
│  │  ┌────────────────────────────────┐  │ │
│  │  │   Plugin Sandbox (WASM)        │  │ │
│  │  │                                │  │ │
│  │  │  • Fuel limit (1M ops)         │  │ │
│  │  │  • No threads, no SIMD         │  │ │
│  │  │  • Isolated memory             │  │ │
│  │  │  • Host function imports       │  │ │
│  │  └────────────────────────────────┘  │ │
│  └──────────────────────────────────────┘ │
└────────────────────────────────────────────┘
```

### Security Constraints

```rust
// Runtime config (from src/middleware/wasm_plugin/runtime.rs)
let mut config = Config::new();
config.async_support(true);
config.consume_fuel(true);        // Prevent infinite loops
config.epoch_interruption(true);   // Kill runaway plugins
config.wasm_multi_memory(false);   // Single memory only
config.wasm_threads(false);        // No threading
config.wasm_simd(false);          // No SIMD
```

### Host Functions Exported to Plugins

Plugins can call these **host functions** from sandbox:

```rust
// Available in WASM as "warpdrive" module
host_log(level: i32, msg: string) -> i32
host_send(conn_id: string, payload: string) -> i32
host_broadcast(stream: string, payload: string) -> i32
host_close(conn_id: string, code: i32, reason: string) -> i32
```

### Plugin Entry Points

Plugins must implement:

**HTTP Protocol:**
```rust
// Plugin must export
wasm_on_request(
    method_ptr: i32, method_len: i32,
    path_ptr: i32, path_len: i32,
    headers_ptr: i32, headers_len: i32,
    body_ptr: i32, body_len: i32
) -> i32  // Return code: 0=allow, 1=block, other=custom
```

**WebSocket Protocol:**
```rust
wasm_on_event(
    event_type: i32,        // 0=connect, 1=message, 2=close
    conn_id_ptr: i32, conn_id_len: i32,
    payload_ptr: i32, payload_len: i32
) -> i32
```

**gRPC Protocol:**
```rust
wasm_on_grpc_call(
    service_ptr: i32, service_len: i32,
    method_ptr: i32, method_len: i32,
    metadata_ptr: i32, metadata_len: i32
) -> i32
```

### Plugin Metadata

Plugins must export metadata functions:

```rust
// Plugin identity
plugin_name() -> i32        // Returns pointer to name
plugin_name_len() -> i32    // Returns name length
plugin_version() -> i32     // Returns pointer to version
plugin_version_len() -> i32 // Returns version length
```

### Route Configuration with Plugins

```toml
# warpdrive.toml
[upstreams.api]
protocol = "http"
host = "127.0.0.1"
port = 3000

[[routes]]
path_prefix = "/api"
upstream = "api"
protocol = "http"
plugin = "plugins/jwt_validator.wasm"  # <-- Plugin loaded per-route
```

---

## Current Plugin Capabilities (Confirmed)

### ✅ What Plugins CAN Do

1. **Access Request Context:**
   - HTTP: method, path, headers, body
   - WebSocket: connection ID, message payloads
   - gRPC: service, method, metadata

2. **Block Requests:**
   - Return non-zero code from `wasm_on_request`
   - Return codes determine action (0=allow, 1=block, custom=app-defined)

3. **Log to Host:**
   - Use `host_log(level, message)` for debugging
   - Logs appear in WarpDrive's tracing output

4. **WebSocket Control:**
   - Send messages to specific connections (`host_send`)
   - Broadcast to all connections in a stream (`host_broadcast`)
   - Close connections (`host_close`)

5. **Isolated Execution:**
   - Fuel limits prevent infinite loops
   - Memory isolation prevents escape
   - Async support for non-blocking operations

### ❓ Unknown / To Be Verified

1. **Can plugins modify request headers?**
   - Current API only reads headers (passed as string)
   - No clear mechanism to return modified headers
   - **Action:** Need to check if return value can encode header changes

2. **Can plugins set response headers (e.g., Set-Cookie)?**
   - No `host_set_response_header` function visible
   - **Action:** May need to add host function for cookie/header injection

3. **Can plugins access/modify response body?**
   - Only request-side hooks visible
   - No `wasm_on_response` entry point
   - **Action:** Check if bidirectional middleware exists

4. **Can plugins store state between requests?**
   - No persistent storage host functions
   - Could use external store (Redis) via custom host function
   - **Action:** Document workaround patterns

5. **Can plugins make HTTP requests?**
   - No `host_http_call` function visible
   - Sandboxed = no network access by default
   - **Action:** Determine if this is intentional or gap

---

## Configuration Management for Immutable Deployments

### Strategy 1: Volume Mount + Restart

```yaml
# docker-compose.yml
services:
  warpdrive:
    image: warpdrive:latest
    volumes:
      - ./warpdrive.toml:/app/warpdrive.toml:ro
      - ./plugins:/app/plugins:ro
    read_only: true
    tmpfs:
      - /tmp
```

**Config Update Process:**
1. Edit `warpdrive.toml` on host
2. Send `SIGHUP` to container → graceful restart
3. New config loaded on restart

### Strategy 2: ConfigMap + Rolling Update (Kubernetes)

```yaml
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: warpdrive-config
data:
  warpdrive.toml: |
    [server]
    http_port = 8080

    [upstreams.api]
    protocol = "http"
    host = "api-service"
    port = 80

---
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: warpdrive
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: warpdrive
        image: warpdrive:v0.3.0
        volumeMounts:
        - name: config
          mountPath: /app/warpdrive.toml
          subPath: warpdrive.toml
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: warpdrive-config
```

**Config Update Process:**
1. `kubectl edit configmap warpdrive-config`
2. `kubectl rollout restart deployment/warpdrive`
3. Rolling update with zero downtime

### Strategy 3: Templated Config + Build-Time Injection

```dockerfile
# Dockerfile
FROM rust:1.75 as builder
WORKDIR /build
COPY . .
RUN cargo build --release

FROM debian:bookworm-slim
COPY --from=builder /build/target/release/warpdrive /usr/local/bin/
COPY warpdrive.toml /app/warpdrive.toml  # Baked in
CMD ["/usr/local/bin/warpdrive", "-c", "/app/warpdrive.toml"]
```

**Config Update Process:**
1. Edit `warpdrive.toml`
2. Build new image: `docker build -t warpdrive:v2 .`
3. Deploy new image

### Strategy 4: Environment Variable Overrides (Partial)

```toml
# warpdrive.toml (template)
[server]
http_port = 8080  # Can be overridden by env var

[upstreams.api]
host = "localhost"  # Override with UPSTREAM_API_HOST env var
port = 3000
```

```bash
# Override at runtime
UPSTREAM_API_HOST=api.internal.svc docker run warpdrive
```

**Limitations:**
- Not all config supports env overrides yet
- Complex structures (routes, plugins) still need file-based config

---

## TLS Certificate Management in Read-Only FS

### Problem
Certs expire → need rotation → can't write to read-only FS

### Solution 1: Volume Mount Cert Directory

```yaml
services:
  warpdrive:
    volumes:
      - /etc/letsencrypt/live:/app/certs:ro
    read_only: true
```

**Rotation Process:**
1. External process (certbot) renews certs in `/etc/letsencrypt/live`
2. Send `SIGHUP` to WarpDrive → reload certs from volume
3. No container restart needed (if graceful reload supported)

**Current Status:** ❓ Need to verify if WarpDrive supports cert reload on signal

### Solution 2: Cert Rotation = New Container

```
1. Certbot renews certs
2. Trigger CI/CD to build new image with new certs
3. Deploy new container (blue-green)
```

**Tradeoff:** Heavyweight but guaranteed consistent

### Solution 3: Cert Volume + Init Container (K8s)

```yaml
initContainers:
- name: cert-sync
  image: certbot-sync
  volumeMounts:
  - name: certs
    mountPath: /certs
  command: ["/sync-certs.sh"]  # Fetch from cert manager

containers:
- name: warpdrive
  volumeMounts:
  - name: certs
    mountPath: /app/certs
    readOnly: true
```

---

## Comparison: WarpDrive vs Aralez

| Aspect | Aralez | WarpDrive |
|--------|--------|-----------|
| **Deployment Model** | Traditional (mutable FS) | Immutable infrastructure |
| **Config Changes** | Hot reload (inotify watch) | New container deployment |
| **Auth** | Built-in (Basic/JWT) | WASM plugin |
| **Middleware** | Core features | Plugin system |
| **Extension** | Fork + rebuild | Load WASM plugin |
| **Service Discovery** | Consul + K8s native | Hook-based (script updates) |
| **TLS Rotation** | File watch + reload | Volume mount + restart |
| **Process Orchestration** | External (systemd) | Built-in supervision |
| **Caching** | None | L1 memory + L2 Redis |
| **Circuit Breaker** | None | Built-in 3-state |
| **Target Use Case** | Edge routing/auth | App-centric orchestration |

---

## Revised Gap Analysis

### NOT Gaps (By Design)

❌ ~~No Hot Reload~~ → Immutable deploys intentional
❌ ~~No Built-in Auth~~ → Use WASM plugins
❌ ~~No File Editing~~ → Volume mounts + restarts

### Actual Gaps (Features Missing)

1. **Service Discovery** (Tier 1)
   - No native Consul/K8s integration
   - **Workaround:** Hook script queries API → generates config → triggers restart
   - **Better Solution:** Pre-start hook that templates config from service registry

2. **Sticky Sessions** (Tier 2)
   - No cookie-based backend affinity
   - **Workaround:** External session store (Redis)
   - **Plugin Potential:** WASM plugin that hashes cookie → selects backend

3. **Plugin Header Modification** (Tier 2)
   - Plugins can READ headers but unclear if they can MODIFY
   - **Investigate:** Check if plugin return value encodes header changes
   - **Potential Gap:** May need new host function `host_set_request_header`

4. **Plugin Response Interception** (Tier 2)
   - Only request-side hooks visible
   - **Gap:** No `wasm_on_response` for response modification
   - **Impact:** Can't inject cookies, modify response headers

5. **Plugin Persistent State** (Tier 3)
   - No host functions for key-value storage
   - **Workaround:** Plugin makes requests to external store
   - **Gap:** Need `host_kv_get/set` for plugin-scoped storage

---

## Recommended Next Steps

### 1. Document Plugin API Fully
- [ ] Create `PLUGIN_API.md` with all host functions
- [ ] Example plugins: JWT validator, rate limiter, auth
- [ ] ABI contract (memory layout, return codes)

### 2. Verify Unknown Capabilities
- [ ] Can plugins modify request headers?
- [ ] Is there response-side plugin support?
- [ ] Can plugins store state?

### 3. Service Discovery Pattern
- [ ] Document hook-based service discovery pattern
- [ ] Example: Consul template hook → generate config → reload
- [ ] K8s example: Init container fetches endpoints → writes config

### 4. Fill Critical Gaps
- [ ] Add `host_set_request_header` if missing
- [ ] Add `wasm_on_response` hook for bidirectional middleware
- [ ] Add `host_kv_get/set` for plugin state

### 5. Deployment Guides
- [ ] Docker Compose example with volume mounts
- [ ] Kubernetes manifests with ConfigMaps
- [ ] CI/CD pipeline for immutable deployments

---

## Conclusion

WarpDrive's immutable, plugin-first architecture is **fundamentally different** from Aralez's hot-reload, built-in approach. It's not missing features—it's solving problems differently:

- **Aralez** = Edge proxy (auth, discovery, dynamic routing)
- **WarpDrive** = App orchestration platform (processes, caching, plugins)

The "gaps" are actually **different paradigms**:
- Hot reload → Immutable deploys
- Built-in auth → Plugin-based auth
- File-based discovery → Hook-based discovery

**This is a strength, not a weakness.**
