# Operator Deploy Guide

This guide covers graceful deployment of the rust-fafb-q server — how to upgrade, restart, or scale server instances without dropping messages, corrupting consumer state, or triggering unnecessary client errors.

## Overview

The server supports zero-downtime deploys through a three-phase lifecycle:

```
Running ──► Draining ──► ShuttingDown ──► Exit
```

- **Running** — normal operation. Accepting connections, processing all frame types.
- **Draining** — planned shutdown in progress. No new connections accepted. Existing clients receive a `GoingAway` notification (if capable) and are given time to finish in-flight work. New subscribes rejected; publishes and acks still accepted.
- **ShuttingDown** — final cleanup. All remaining connections force-closed. Engine and storage flushed. Process exits.

Deploys are distinguished from crashes at the wire level: clients that receive `GoingAway` know to reconnect calmly. Clients that see an unexpected TCP close should assume failure and alert.

## Configuration Reference

All deploy-related config is in the `[deploy]` section:

```toml
[deploy]
# How long to wait in drain mode for clients to disconnect voluntarily.
# Clients receive GoingAway and should disconnect within this window.
# After this expires, remaining connections are force-closed.
# Default: 30 seconds. Set higher for large deployments (5,000+ subscribers).
# Set to 0 for immediate force-close (same behavior as pre-14 releases).
drain_timeout_secs = 30

# How long after drain to wait for engine/storage shutdown (fsync, cleanup).
# This is independent of drain — drain is about client connections, shutdown
# is about storage flush.
# Default: 10 seconds.
shutdown_timeout_secs = 10

# Reason code sent in GoingAway frames. Informational — client behavior is
# identical regardless. Values: "upgrade" (default), "maintenance", "shutdown".
going_away_reason = "upgrade"

# API key for admin endpoints (/admin/*). Must be >= 24 characters.
# If not set, admin endpoints return 403.
# Can also be set via FAFB_ADMIN_API_KEY environment variable.
# admin_api_key = "your-admin-key-here-minimum-24-chars"
```

### Tuning Guidance

| Scenario | `drain_timeout_secs` | `shutdown_timeout_secs` |
|----------|---------------------|------------------------|
| Dev/test restart | 0–5 | 5 |
| Small deployment (<100 clients) | 10–15 | 10 |
| Medium deployment (100–1,000 clients) | 30 | 10 |
| Large deployment (1,000–10,000 clients) | 60–120 | 15 |
| Cold-read heavy consumers | 60+ | 10 |

If clients are doing cold reads (consuming from disk-backed segments), they may need more time to ack in-flight messages. Watch `fafb_drain_duration_seconds` to tune the timeout based on actual drain behavior.

## Single-Instance Deploy (SIGTERM)

The simplest deploy path — one server, one process:

```bash
# 1. Send SIGTERM to the running server
kill $SERVER_PID

# The server will:
#   a. Force-flush all buffered writes to disk
#   b. Transition to Draining — stop accepting new connections
#   c. Send GoingAway to capability-aware clients
#   d. Send Error(ServerDraining) to legacy clients
#   e. Wait drain_timeout_secs for clients to disconnect
#   f. Force-close remaining connections
#   g. Flush engine/storage, exit

# 2. Start the new version
./rust-fafb-q --config config.toml
```

What happens at each stage:

1. **Force-flush** — all buffered writes are synced to disk. This closes the durability gap between the last batch flush and the signal.
2. **GoingAway** — capability-aware clients receive a `GoingAway` frame with the configured reason and remaining drain time. They should ack in-flight messages, unsubscribe, and disconnect.
3. **Legacy clients** — clients that didn't negotiate `CAP_GOING_AWAY` receive `Error { ServerDraining }` and disconnect immediately.
4. **Drain wait** — the server logs progress every 5 seconds: "draining: N connections remaining". If all connections close before the timeout, it proceeds immediately.
5. **Shutdown** — remaining connections are force-closed, delivery tasks aborted, storage fsynced.

Clients reconnect to the same address after the new version starts. Offset continuity is maintained — clients that tracked their last acked offset (via `Deliver.offset`) can resume with `SubscribeAt` or `Reconnect`.

## Rolling Deploy (Multi-Instance)

For multi-instance deployments behind a load balancer:

```
1. Mark instance N as "draining" via readiness probe
2. LB stops routing new connections to instance N
3. Instance N drains existing connections
4. Stop instance N, start new version
5. New instance N becomes ready (/health/ready → 200)
6. LB routes traffic to new instance N
7. Repeat for instances N+1, N+2, ...
```

The key mechanism is `/health/ready`:
- Returns `200` when initialized and Running
- Returns `503` when initializing, draining, or shutting down
- LBs that check this endpoint automatically stop routing to draining instances

### Kubernetes

```yaml
apiVersion: apps/v1
kind: Deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 45  # drain_timeout + shutdown_timeout + buffer
      containers:
        - name: fafb-q
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 5
          startupProbe:
            httpGet:
              path: /health/ready
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
```

> **Important:** Set `terminationGracePeriodSeconds` to at least `drain_timeout_secs + shutdown_timeout_secs + 5`. Kubernetes sends SIGTERM, then waits this long before SIGKILL. If the grace period is shorter than the drain timeout, the process is killed mid-drain.

### ECS

```json
{
  "healthCheck": {
    "command": ["CMD-SHELL", "curl -sf http://localhost:8080/health/ready || exit 1"],
    "interval": 5,
    "timeout": 3,
    "retries": 3,
    "startPeriod": 30
  },
  "stopTimeout": 45
}
```

### HAProxy

```
backend fafb-q
    option httpchk GET /health/ready
    http-check expect status 200
    server fafb1 10.0.0.1:8080 check inter 5s fall 2 rise 3
    server fafb2 10.0.0.2:8080 check inter 5s fall 2 rise 3
```

### nginx upstream

```nginx
upstream fafb_q {
    server 10.0.0.1:8080 max_fails=2 fail_timeout=10s;
    server 10.0.0.2:8080 max_fails=2 fail_timeout=10s;
}
```

With nginx, use an external health checker or the commercial `health_check` directive to monitor `/health/ready`.

## HTTP Drain API

For scripted or orchestrated deploys, use the admin drain endpoint instead of (or before) SIGTERM:

```bash
# 1. Configure admin API key
export FAFB_ADMIN_API_KEY="your-admin-key-here-minimum-24-chars"

# 2. Trigger drain with optional overrides
curl -X POST http://localhost:8080/admin/drain \
  -H "Authorization: Bearer $FAFB_ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"reason": "upgrade", "timeout_secs": 60}'

# Response: {"status":"draining","active_connections":4523,"deadline":"2026-03-21T15:30:00Z"}

# 3. Monitor drain progress
curl http://localhost:8080/admin/drain/status \
  -H "Authorization: Bearer $FAFB_ADMIN_API_KEY"

# Response: {"state":"draining","active_connections":127,"drain_started_at":"...","drain_deadline":"...","connections_drained":4396}

# 4. Wait for readiness to flip
while curl -sf http://localhost:8080/health/ready > /dev/null 2>&1; do sleep 1; done

# 5. Server is drained — safe to stop
kill $SERVER_PID
```

Use cases:
- Deploy scripts that need to drain before sending SIGTERM
- Canary deploys that drain the old instance before shifting traffic
- Operator dashboards that trigger drain without SSH access

## Automated Upgrade Script

For single-instance deployments, `scripts/rolling-upgrade.sh` automates the full upgrade sequence:

```bash
# Download latest and upgrade
FAFB_ADMIN_API_KEY=mykey ./scripts/rolling-upgrade.sh

# Use a specific binary
FAFB_ADMIN_API_KEY=mykey ./scripts/rolling-upgrade.sh --binary /path/to/fafb-q

# Force completion even if drain times out
./scripts/rolling-upgrade.sh --force
```

The script handles: preflight validation, drain trigger via admin API, shutdown wait, binary swap, restart, and health verification. See `scripts/rolling-upgrade.sh --help` for all options.

For multi-instance rolling deploys, use your orchestrator (Kubernetes, ECS) with the health endpoints documented above.

## Monitoring During Deploy

### Key Metrics

| Metric | Type | What to Watch |
|--------|------|---------------|
| `fafb_server_state` | gauge | 0=running, 1=draining, 2=shutting_down. Alert if stuck in draining for longer than `drain_timeout + 30s`. |
| `fafb_drain_duration_seconds` | histogram | Time from drain start to all connections closed. Trend this to tune `drain_timeout_secs`. |
| `fafb_going_away_sent_total` | counter | GoingAway frames sent. Should spike once per deploy, then stop. |
| `fafb_reconnect_total{status}` | counter | Reconnect frames processed. Labels: `success`, `partial`, `auth_failed`. High `partial` or `auth_failed` rates indicate client misconfiguration. |
| `fafb_reconnect_subscriptions` | histogram | Subscriptions per Reconnect frame. Helps size Reconnect processing capacity. |
| `fafb_connections_active` | gauge | Active TCP connections. Should drop to 0 during drain. |

### Alerting Recommendations

```yaml
# Alert if server is stuck draining
- alert: FafbDrainStuck
  expr: fafb_server_state == 1
  for: 2m
  annotations:
    summary: "fafb-q stuck in drain mode for >2 minutes"

# Alert on high reconnect failure rate
- alert: FafbReconnectFailures
  expr: rate(fafb_reconnect_total{status="auth_failed"}[5m]) > 0.1
  annotations:
    summary: "fafb-q seeing Reconnect auth failures after deploy"
```

## Troubleshooting

### Clients not reconnecting after deploy

**Symptoms:** `fafb_connections_active` stays at 0 after new server starts.

**Causes:**
- Clients don't implement `GoingAway` handling — check if clients negotiate `CAP_GOING_AWAY`
- Clients reconnecting to old address/port — verify DNS/service discovery updated
- Reconnect backoff too aggressive — check client SDK reconnect settings

**Fix:** Ensure client SDKs send `capabilities: CAP_GOING_AWAY | CAP_RECONNECT` in Connect. Legacy clients will reconnect via standard error-recovery paths but may take longer.

### Drain timeout too short

**Symptoms:** `fafb_drain_duration_seconds` consistently hits the configured timeout. Connections are force-closed.

**Causes:**
- Cold-read consumers take too long to finish in-flight work
- Clients ignoring GoingAway (not disconnecting voluntarily)
- `drain_timeout_secs` set too low for the fleet size

**Fix:** Increase `drain_timeout_secs`. Monitor `fafb_drain_duration_seconds` to find the right value. For cold-read heavy workloads, consider 60–120 seconds.

### Thundering herd after restart

**Symptoms:** New server instance overwhelmed with connection attempts immediately after start. `fafb_connections_active` spikes, potential `ResourceExhausted` errors.

**Causes:**
- All clients reconnecting simultaneously without jitter
- `drain_timeout_secs` too short — clients all start reconnecting at the same time

**Fix:** Ensure client SDKs use the recommended jitter formula: `deadline_ms + uniform_random(0, min(5000, deadline_ms / 2))`. Increase `drain_timeout_secs` to widen the jitter window. Increase `max_connections` if needed for the spike.

### Health probe failing during startup

**Symptoms:** `/health/ready` returns 503 for an extended period after process start.

**Causes:**
- `engine.init()` replaying large datasets from disk (thousands of topics, millions of messages)
- Storage I/O bottleneck during topic replay

**Fix:** This is expected behavior — the server isn't ready until replay completes. Increase the startup probe `failureThreshold` in Kubernetes. Monitor replay duration via server logs ("topic replay completed in Ns"). Consider reducing the amount of retained data or using faster storage.
