# Rolling Upgrade Script Design

Convenience script that orchestrates a zero-downtime upgrade of a running fafb-q server: drain, stop, swap binary, restart, verify health.

## Usage

```bash
# Download latest and upgrade local server
FAFB_ADMIN_API_KEY=mykey ./scripts/rolling-upgrade.sh

# Use a pre-staged binary
FAFB_ADMIN_API_KEY=mykey ./scripts/rolling-upgrade.sh --binary /tmp/fafb-q-new

# Non-default admin endpoint
./scripts/rolling-upgrade.sh --admin-url http://10.0.0.5:8080 --admin-key mykey

# Force-kill if drain/shutdown times out
./scripts/rolling-upgrade.sh --force

# Non-systemd server with explicit config
./scripts/rolling-upgrade.sh --config /etc/fafb-q/config.toml
```

## Requirements

- POSIX `sh` compatible — no bashisms. Same constraints as `install.sh`.
- Requires `curl` for admin API calls and binary download.
- Linux only (x86_64 or aarch64). Same platform constraints as `install.sh`.
- Uses `sudo` only when needed for binary installation (same pattern as `install.sh`).
- Cleans up temporary files on exit (success or failure) via trap on EXIT/INT/TERM.

## CLI Flags

| Flag | Default | Description |
|------|---------|-------------|
| `--admin-url URL` | `http://localhost:8080` | Admin/health endpoint base URL |
| `--admin-key KEY` | `$FAFB_ADMIN_API_KEY` env var | Admin API key for `/admin/drain` |
| `--binary PATH` | _(download latest)_ | Path to pre-staged binary. Skips download. |
| `--force` | off | SIGKILL the old process if drain/shutdown times out |
| `--drain-timeout SECS` | 30 | How long to wait for the server process to exit after triggering drain. This is the script's poll timeout, not the server's `drain_timeout_secs` config. |
| `--config PATH` | _(none)_ | Config file path passed to server on restart. Required in non-systemd mode. |
| `--help` | — | Print usage information and exit. |

If `--admin-key` is not provided and `$FAFB_ADMIN_API_KEY` is not set, the script exits with an error before touching anything.

## Script Behavior

### Step 1: Preflight Checks

Validate everything before touching the running server.

- Verify `curl` is available.
- Verify admin key is set (flag or env var).
- Verify server is reachable: `GET $ADMIN_URL/health/ready` returns 200.
- If `--binary` is provided, verify the file exists and is executable.
- If `--binary` is not provided, download the latest binary (see Step 3).
- Run the new binary with `--help` to verify it executes (exit code 0). Note: `--version` does not currently exist on the binary. Adding `#[command(version)]` to the Cli struct is a prerequisite tracked separately. Once added, this check should switch to `--version` for a cleaner version capture.

If any preflight check fails, exit with a clear error. Nothing is changed.

### Step 2: Resolve Process Management

Detect how the running server is managed:

```
if systemctl list-unit-files fafb-q.service exists → systemd mode
else → direct PID mode (pidof fafb-q or pgrep -x fafb-q)
```

**Systemd mode:**
- The drain API (Step 4) triggers the full shutdown sequence — the server exits on its own after drain completes. `systemctl stop` is NOT needed in the normal path.
- Force-kill (`--force`): `systemctl kill -s KILL fafb-q` if the process doesn't exit within the timeout.
- Start: `systemctl start fafb-q`
- Status: `systemctl is-active fafb-q`

**Direct PID mode:**
- Find PID via `pidof fafb-q` or `pgrep -x fafb-q`.
- If no PID found, exit with error ("no running fafb-q process found").
- Resolve the binary path from `/proc/$PID/exe` (for install target). Note: this requires running as the same user or root. If it fails, fall back to `/usr/local/bin/fafb-q` default.
- The drain API triggers shutdown; poll `kill -0 $PID` until process exits.
- Force-kill (`--force`): `kill -9 $PID` if the process doesn't exit within the timeout.
- Start: `nohup $BINARY_PATH serve --config $CONFIG_PATH > /dev/null 2>&1 &` (requires `--config`). Uses `nohup` instead of `disown` for POSIX sh compatibility.
- If `--config` is not set in direct PID mode, the script installs the binary but exits after step 6 with a message: "Binary installed. Start the server manually." This avoids guessing config paths.

### Step 3: Download Binary (if no `--binary`)

Same logic as `install.sh`:
1. Detect architecture via `uname -m` → target triple.
2. Download `fafb-q-latest-{target}.tar.gz` from `https://bitbucket.org/thermsio/fafb-q/downloads/` (same `BASE_URL` as `install.sh`) to a temp directory.
3. Extract with `tar xzf --strip-components=1`.
4. Verify the binary exists and is executable.

### Step 4: Trigger Drain

```bash
http_code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "$ADMIN_URL/admin/drain" \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"reason": "upgrade"}')
```

Handle the response:
- **200**: drain accepted. Proceed.
- **409**: server is already draining. Print warning ("Server is already draining, proceeding to wait for shutdown") and skip to Step 5.
- **Any other code**: exit with error.

After triggering (or on 409), poll `GET $ADMIN_URL/health/ready` every 2 seconds until it returns 503 (confirming the server has entered drain mode). If it doesn't flip within 10 seconds, print a warning but continue (the drain was accepted; the ready endpoint may already be 503 by the time we poll).

Note: `/admin/drain/status` provides richer progress info but requires the admin auth header. The script uses `/health/ready` (no auth needed) for simplicity.

### Step 5: Wait for Shutdown

Poll until the server process exits:

- **Systemd mode:** poll `systemctl is-active fafb-q` until it returns `inactive`.
- **Direct PID mode:** poll `kill -0 $PID` until it fails (process gone).

Poll interval: 2 seconds. Total timeout: `--drain-timeout` value (default 30s).

**If timeout expires:**
- Without `--force`: exit with error. The server is still running (possibly mid-drain). Print: "Drain timed out. Server is still running. Use --force to kill, or increase --drain-timeout."
- With `--force`: send SIGKILL to the process (or `systemctl kill -s KILL fafb-q`). Wait up to 5 seconds for it to die. If still alive, exit with error.

### Step 6: Install Binary

Install the new binary to the same path as the old one:

- **Systemd mode:** default to `/usr/local/bin/fafb-q` (most common systemd unit setup).
- **Direct PID mode:** use the path resolved from `/proc/$PID/exe` in step 2.
- Override path if the old binary was at a non-standard location.

Uses `install -m 755` with conditional `sudo` (same `run()` pattern as `install.sh`). Creates the target directory with `mkdir -p` if needed.

### Step 7: Start and Verify

- **Systemd mode:** `systemctl start fafb-q`. After starting, check `systemctl is-active fafb-q`. If it returns `failed`, print the output of `systemctl status fafb-q` and exit with error ("Server failed to start. Check logs with: journalctl -u fafb-q").
- **Direct PID mode (with `--config`):** `nohup $BINARY_PATH serve --config $CONFIG_PATH > /dev/null 2>&1 &` (uses `nohup` for POSIX sh compatibility — `disown` is a bashism).
- **Direct PID mode (no `--config`):** skip start, print "Binary installed. Start the server manually."

After starting, poll `GET $ADMIN_URL/health/ready` every 2 seconds until it returns 200. Timeout: 60 seconds (accounts for large dataset replay during `engine.init()`).

If health check passes, print the new version and success message. If it times out, print a warning: "Server started but /health/ready not yet 200. It may still be initializing."

## Error Handling

- `set -eu` at the top.
- Preflight failures exit before any side effects.
- Drain/shutdown failures leave the old server running (safe default).
- The `--force` flag is the only way to escalate past a timeout.
- Temp dir cleanup via `trap 'rm -rf "$tmpdir"' EXIT INT TERM`.

## Output

Plain text, no colors. Each step prints a status line.

```
Preflight: server reachable at http://localhost:8080
Preflight: downloading latest fafb-q for x86_64-unknown-linux-musl...
Preflight: new binary version: fafb-q 0.2.0
Draining server...
Drain accepted. Waiting for server to shut down...
Server stopped (was PID 12345).
Installing fafb-q 0.2.0 to /usr/local/bin/fafb-q...
Starting server via systemctl...
Health check: /health/ready returned 200
fafb-q 0.2.0
Upgrade complete.
```

## Inline Documentation

The script must include:

1. **Header comment block** explaining what the script does, usage examples, prerequisites (admin key, running server), and a link to `docs/deploy.md` for the full deploy guide.
2. **Per-step comments** explaining the "why" at each stage — especially the drain→poll→shutdown→install→start sequence and why ordering matters.
3. **Flag parsing section** with a comment documenting each flag's purpose and default.
4. **Process management detection** with a comment explaining the systemd vs. direct PID distinction and why both are needed.

## Documentation Updates

### `docs/deploy.md`

Add a new section after "HTTP Drain API" titled "Automated Upgrade Script":

```markdown
## Automated Upgrade Script

For single-instance deployments, `scripts/rolling-upgrade.sh` automates the full upgrade sequence:

\```bash
# Download latest and upgrade
FAFB_ADMIN_API_KEY=mykey ./scripts/rolling-upgrade.sh

# Use a specific binary
FAFB_ADMIN_API_KEY=mykey ./scripts/rolling-upgrade.sh --binary /path/to/fafb-q

# Force completion even if drain times out
./scripts/rolling-upgrade.sh --force
\```

The script handles: preflight validation, drain trigger via admin API, shutdown wait, binary swap, restart, and health verification. See `scripts/rolling-upgrade.sh --help` for all options.

For multi-instance rolling deploys, use your orchestrator (Kubernetes, ECS) with the health endpoints documented above.
```

### `README.md`

Add a "Live upgrade" section after the install section, or append to the existing install section:

```markdown
### Live upgrade

Upgrade a running server with zero downtime:

\```bash
FAFB_ADMIN_API_KEY=mykey curl -fsSL https://bitbucket.org/thermsio/fafb-q/downloads/rolling-upgrade.sh | sh
\```

Or with a local copy:

\```bash
FAFB_ADMIN_API_KEY=mykey ./scripts/rolling-upgrade.sh
\```

See `scripts/rolling-upgrade.sh --help` for options (custom admin URL, pre-staged binary, force mode). Full deploy documentation: [`docs/deploy.md`](docs/deploy.md).
```

### `CLAUDE.md`

Add `scripts/rolling-upgrade.sh` to the module map.

### `docs/release-checklist.md`

No changes needed — the script is committed to the repo and uploaded via the pipeline like `install.sh`.

## Pipeline Upload

Add `rolling-upgrade.sh` to the Bitbucket Downloads upload in the `&upload-downloads` step, same pattern as `install.sh` (see `bitbucket-pipelines.yml` lines 79-85 for the template). Add a matching block:

```yaml
          - |
            echo "Uploading rolling-upgrade.sh..."
            curl -X POST "$UPLOAD_URL" \
              --user "$ATLASSIAN_ACCOUNT_EMAIL:$ATLASSIAN_API_TOKEN" \
              --form files=@"scripts/rolling-upgrade.sh" \
              --fail --show-error
```

## File Location

- Source: `scripts/rolling-upgrade.sh`
- Published: Bitbucket Downloads (same upload pattern as `install.sh`)

## Prerequisites

- **`--version` flag on the fafb-q binary**: Currently missing. Add `#[command(version)]` to the `Cli` struct in `src/main.rs`. This is a one-line change that should be done before or alongside this script. Until then, the preflight smoke test uses `--help` instead. This also fixes a latent bug in `install.sh` which calls `--version` as a smoke test.

## Out of Scope

- Multi-instance rolling upgrades (use Kubernetes/ECS — documented in `docs/deploy.md`)
- Version comparison (does not check if the new binary is actually newer)
- Rollback (operator responsibility — keep the old binary, re-run with `--binary`)
- Checksum/signature verification (same as `install.sh` — no checksums published today)
- Windows/macOS support
