# Phase 14: Graceful Deployment & Hot Upgrade

## Goal

Enable zero-downtime deployments of the fafb-q server while thousands of clients are actively publishing and consuming. A deploy must not silently drop messages, corrupt consumer group state, or force clients into error-recovery paths that are indistinguishable from a real failure. The protocol itself must carry the semantics of "I am going away on purpose" so that every client — current and future — can handle upgrades as a first-class event, not an error to be guessed at.

## Guiding Principles

- **Deploys are not errors.** A planned shutdown must be distinguishable at the wire level from a crash. Clients that receive a deploy signal should reconnect calmly; clients that see a TCP RST should assume failure and alert.
- **Offset continuity is sacred.** A subscriber at offset N before a deploy must resume at offset N after reconnecting. No implicit rewind, no silent gap. The protocol must make this easy. Clients MUST track the `offset` field from every `Deliver` frame they acknowledge — this is a protocol-level requirement, not a suggestion.
- **The server controls the timeline.** The server decides when to stop accepting, how long to drain, and when to force-close. Clients react to the server's signals, not the other way around.
- **Single-instance deploys must work.** Not every deployment has a load balancer or multiple instances. The protocol must support the degenerate case: one server, SIGTERM, clients reconnect to the same address after the new version starts.
- **Additive protocol changes only.** New frame types must not break v1 clients that don't understand them. Unknown frame types in the current codec are treated as errors — so capability negotiation must land BEFORE any new frame types, ensuring the server never sends a frame a client can't handle.
- **Ship the minimum that makes deploys safe, then iterate.** The core deploy story (drain + GoingAway + health probes) ships first. Protocol optimizations (Reconnect frame) ship after the foundation is validated in production.

## Current State

> Note: this project is still in development and it has not yet been published - no backwards compat concerns.

### What exists

| Capability | Implementation | Gap |
|---|---|---|
| Signal handling | SIGTERM/SIGINT → `watch::channel` broadcast | Works, but shutdown is binary — no drain phase |
| Connection goodbye | Sends `Frame::Error { code: InternalError, message: "server shutting down" }` | Indistinguishable from a real internal error |
| Shutdown timeout | 10-second hard deadline, then `process::exit(1)` | No configurable drain period separate from cleanup timeout |
| Health endpoint | `GET /health` → always 200 | No readiness state; load balancers can't distinguish "starting" from "draining" |
| Protocol version | `Frame::Connect { version: u8 }` — hard reject if != 1 | No negotiation, no feature flags, no forward compatibility |
| Offset resume | `Frame::SubscribeAt { topic, offset, group }` (0x0E) | Client must track own offset and re-subscribe manually per topic |
| Consumer group state | Server-side offset tracking, inflight map, round-robin | All state is in-memory — lost on shutdown, rebuilt from storage on restart |
| Inflight timeout | 300s default, DLQ after 5 redeliveries | Good — but no accelerated drain of inflight on shutdown |

### What's missing

1. **Capability negotiation** — replace hard version check with feature flags so new frames don't break old clients (must land first)
2. **`GoingAway` frame** — a dedicated, non-error signal that tells clients "disconnect gracefully and reconnect soon"
3. **Drain mode** — server state machine with `Running → Draining → ShuttingDown` transitions
4. **Readiness endpoint** — `/health/ready` that returns 503 during drain and before init completes
5. **Drain timeout config** — separate from shutdown timeout; how long to wait for clients to leave voluntarily
6. **Inflight acceleration** — on drain, reduce inflight timeout so unacked messages are resolved quickly
7. **Storage flush at drain entry** — force-flush batch buffer before sending GoingAway to guarantee durability
8. **Reconnect frame** — client tells server "I was here before, here are my subscriptions and offsets" in a single round-trip (optimization, ships after core)
9. **Documentation** — deploy runbook, client SDK reconnect guide, protocol changelog

---

## Phases

### Phase 14.1 — Drain State Machine, Health Endpoints & Capability Negotiation

**Goal:** Lay the complete foundation: server lifecycle state machine, readiness-aware health probes, and capability negotiation in the Connect/ConnectAck handshake. These three concerns are tightly coupled — the state machine drives health responses, and capabilities must exist before any new frame types can be safely sent — so they ship as one unit.

#### Server state enum

```rust
// src/server/state.rs (new file)

use std::sync::atomic::{AtomicBool, AtomicU8, Ordering};
use tokio::sync::watch;

#[repr(u8)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ServerState {
    /// Normal operation. Accepting connections, processing all frames.
    Running = 0,
    /// Draining. No new connections accepted. Existing connections receive
    /// GoingAway (if capable) and are given time to finish. Publishes still
    /// accepted (so in-flight work completes). New subscribes rejected.
    Draining = 1,
    /// Final shutdown. All connections force-closed. Engine and storage
    /// flushed and shut down.
    ShuttingDown = 2,
}

pub struct ServerLifecycle {
    state: AtomicU8,
    initialized: AtomicBool,       // false until engine.init() completes
    active_connections: AtomicU64,  // tracks connections for drain progress
    notify: watch::Sender<ServerState>,
    // Drain metadata (set on transition to Draining)
    drain_started_at: Mutex<Option<Instant>>,
    connections_at_drain_start: AtomicU64,
}

impl ServerLifecycle {
    pub fn is_ready(&self) -> bool {
        self.initialized.load(Ordering::Acquire)
            && self.state.load(Ordering::Acquire) == ServerState::Running as u8
    }

    pub fn is_draining(&self) -> bool {
        self.state.load(Ordering::Acquire) >= ServerState::Draining as u8
    }
}
```

> **Note:** `AtomicU8` for lock-free reads on the hot path (every frame dispatch checks `is_draining()`). The `watch::Sender` broadcasts state transitions to all tasks that need to react (TCP accept loop, HTTP server, connection handlers, background tasks). This replaces the existing `watch::channel(false)` shutdown signal — the old bool channel is subsumed by the state enum.

#### Capability negotiation

Capabilities must land in this phase — before any new frame types — because without them, the server has no way to know whether a connected client can handle `GoingAway`. Sending an unknown frame type (0x10) to a v1 client causes a codec error and an ugly disconnect with error logs. With 5,000 clients, that's 5,000 spurious error logs on every deploy, masking real failures.

**Design: Capability flags in Connect/ConnectAck**

```
Frame::Connect (updated wire format):
  [u8  version]              — still 1
  [u16 namespace_length]
  [u8* namespace]
  [u16 api_key_length]
  [u8* api_key]
  [u32 capabilities]         — NEW: bitmask of client capabilities (0 if not present)

Frame::ConnectAck (updated wire format):
  [u8  status]
  [u16 message_length]
  [u8* message]
  [u32 server_capabilities]  — NEW: bitmask of capabilities the server agrees to use
```

> **Backward compatibility:** The frame length prefix tells the decoder exactly how many bytes are in the payload. After parsing version + namespace + api_key, if 4+ bytes remain, parse capabilities as `u32`. If 0 bytes remain, capabilities = 0 (legacy v1 client). This is safe because the decoder always knows the total frame size from the envelope.

**Capability bits — plain constants, no crate needed:**

```rust
// src/codec/frame.rs

pub type Capabilities = u32;

/// Client/server understands GoingAway frame (0x10).
pub const CAP_GOING_AWAY: Capabilities = 1 << 0;
/// Client/server understands Reconnect/ReconnectAck (0x11, 0x12).
pub const CAP_RECONNECT: Capabilities = 1 << 1;
// Bits 2-31 reserved for future capabilities.

/// All capabilities this server version supports.
pub const SERVER_CAPABILITIES: Capabilities = CAP_GOING_AWAY | CAP_RECONNECT;
```

> **Note:** No `bitflags` crate. Two bits don't justify a dependency. Plain `u32` constants with bitwise ops are clearer, have zero cost, and are trivially extensible. If the capability space grows past ~8 bits in a future plan, reconsider.

**Negotiation protocol:**

1. Client sends `Connect { version: 1, ..., capabilities: CAP_GOING_AWAY }`
2. Server intersects client capabilities with its own: `agreed = client_caps & SERVER_CAPABILITIES`
3. Server sends `ConnectAck { ..., server_capabilities: agreed }`
4. Both sides use only the agreed capabilities for this session
5. If a client doesn't send capabilities (v1 legacy), `agreed = 0` — no new features, no regression

**Per-session capability tracking:**

The `AuthenticatedState` struct gets a `capabilities: Capabilities` field set during the Connect handshake. The connection handler checks this field when deciding whether to send `GoingAway` or fall back to the legacy `Error { InternalError }` on drain.

#### Health endpoints

```
GET /health/live
  → Always 200 while the process is running.
  → Response: { "status": "alive" }
  → No auth required.
  → Use case: Kubernetes liveness probe, systemd watchdog.

GET /health/ready
  → 200 when the server is fully initialized and in Running state.
  → 503 when:
     - Server is still initializing (engine.init() not complete)
     - Server is in Draining state
     - Server is in ShuttingDown state
  → Response: { "status": "ready"|"initializing"|"draining"|"shutting_down" }
  → No auth required.
  → Use case: Kubernetes readiness probe, load balancer health check.

GET /health (existing, changed to readiness-aware)
  → Alias for /health/ready (same behavior as new readiness endpoint).
  → Existing clients that check /health get correct drain behavior for free.
```

> **Note:** Changing `/health` from always-200 to readiness-aware is a **breaking change** for any monitoring that assumes `/health` always returns 200. This is the correct behavior — a draining server must NOT receive new traffic from a load balancer. The default is readiness-aware. If an operator needs the old behavior during migration, they can check `/health/live` instead.

**Readiness gating on startup:**

The HTTP server must start BEFORE `engine.init()` so that health probes work during startup. The TCP server waits until `initialized == true` before accepting. Business routes (publish/query/info) return 503 until initialized.

```
Process starts
  ├─ HTTP server binds (required for health probes to work)
  ├─ /health/live → 200 (immediately)
  ├─ /health/ready → 503 "initializing"
  │
  ├─ storage.init() runs
  ├─ engine.init() runs (topic replay — can take seconds for large datasets)
  │
  ├─ Initialization complete, lifecycle.initialized = true
  ├─ /health/ready → 200 "ready"
  ├─ TCP server starts accepting connections
  │
  ... normal operation ...
  │
  ├─ SIGTERM received → lifecycle transitions to Draining
  ├─ /health/ready → 503 "draining"
  ├─ TCP server stops accepting
  │
  ... drain timeout ...
  │
  ├─ /health/ready → 503 "shutting_down"
  └─ Process exits
```

#### Config additions

```toml
[deploy]  # originally planned as [server], implemented as [deploy] — see implementation notes
# How long to wait in drain mode before force-closing connections.
# During this window, clients receive GoingAway and should disconnect voluntarily.
# After this expires, remaining connections are force-closed and shutdown proceeds.
# Default: 30 seconds. Set higher for large deployments with many consumers.
# Set to 0 for immediate force-close (same behavior as pre-14 releases).
drain_timeout_secs = 30

# How long after drain completes to wait for engine/storage shutdown.
# This is the existing SHUTDOWN_TIMEOUT, now configurable.
# Default: 10 seconds.
shutdown_timeout_secs = 10

# Reason code sent in GoingAway frames. Informational — clients behave the
# same regardless, but SDKs can log it for operator visibility.
# Values: "upgrade" (default), "maintenance", "shutdown"
going_away_reason = "upgrade"
```

> **Note:** `drain_timeout_secs = 30` is the key knob. For a deployment with 5,000 subscribers doing cold reads, 30s gives clients time to ack inflight, unsubscribe, and disconnect. For a quick dev restart, set to 0 (immediate force-close, same as today). The separation between drain timeout and shutdown timeout is important: drain is about client connections, shutdown is about storage flush. They are independent concerns with independent scaling characteristics.

> **Removed:** `accept_publishes_during_drain` config flag. Publishes are always accepted during drain. Rejecting publishes stalls the publish-subscribe pipeline mid-drain — exactly what we're trying to avoid. Every in-flight publish should complete so the subscriber can ack it before disconnecting. The added config flag would require its own test matrix and error handling in client SDKs for marginal value. If a future use case demands it, it can be added then.

#### Revised shutdown sequence

```
SIGTERM received
    │
    ├─ 1. Force-flush storage batch buffer
    │     └─ Ensures all buffered writes are durable before clients disconnect.
    │        Without this, up to 100ms / 4MB of writes could be lost if the
    │        process crashes during drain.
    │
    ├─ 2. Transition to Draining
    │     ├─ TCP accept loop: stop accepting new connections
    │     ├─ HTTP server: /health/ready returns 503
    │     ├─ All connection handlers: check per-session capabilities
    │     │   ├─ CAP_GOING_AWAY set: send GoingAway frame
    │     │   └─ CAP_GOING_AWAY not set: send Error { InternalError, "server shutting down" }
    │     └─ Session state machine: reject new Subscribe/Connect frames
    │         with ErrorCode::ServerDraining (new error code)
    │
    ├─ 3. Wait for drain_timeout_secs
    │     ├─ Clients disconnect voluntarily after receiving GoingAway
    │     ├─ Server tracks active connection count via lifecycle
    │     ├─ If all connections close before timeout: proceed immediately
    │     └─ Log progress: "draining: N connections remaining" every 5s
    │
    ├─ 4. Transition to ShuttingDown
    │     ├─ Force-close all remaining connections (send Error frame)
    │     ├─ Abort all delivery tasks
    │     ├─ Call manager.shutdown()
    │     └─ Call engine.shutdown() (flush storage)
    │
    └─ 5. Exit
          ├─ If shutdown completes within shutdown_timeout_secs: exit 0
          └─ If timeout: log error, exit 1
```

> **Note:** Step 1 (force-flush) is new. The existing batch flush runs every 100ms, but during the window between the last flush and the SIGTERM, unflushed data exists only in memory. If a client reconnects to a new instance that reads from the same disk, that data must be there. One explicit `engine.force_flush()` at drain entry closes this durability gap.

> **Note:** Step 3 has an early-exit optimization: if all clients disconnect before the drain timeout, skip waiting. This makes single-client dev scenarios fast. The 5-second progress log prevents operators from staring at a silent process wondering if it's stuck.

#### Inflight acceleration during drain

When the server enters `Draining`:
- Reduce the inflight timeout from 300s to `min(30s, drain_timeout_secs / 2)`
- This ensures unacked messages are resolved (redelivered or DLQ'd) before shutdown
- Do NOT change the max redelivery count — just the timeout per attempt

> **Note:** Without this, a client that disconnects without acking leaves messages in limbo for 5 minutes. With a 30s drain, those messages would be abandoned. Accelerating the timeout ensures they're requeued (and potentially picked up by another group member who's still connected during drain) or DLQ'd.

#### Work items

- [x] Create `src/server/state.rs` with `ServerState` enum and `ServerLifecycle` struct (including `initialized`, `active_connections`, drain metadata)
- [x] Add `Capabilities` type alias, `CAP_GOING_AWAY`, `CAP_RECONNECT`, `SERVER_CAPABILITIES` constants to `src/codec/frame.rs`
- [x] Update `Frame::Connect` to include `capabilities: Capabilities` field
- [x] Update `Frame::ConnectAck` to include `server_capabilities: Capabilities` field
- [x] Encoder: append capabilities `u32` to Connect and ConnectAck
- [x] Decoder: detect trailing capabilities bytes in Connect (backward-compatible parse via remaining frame bytes)
- [x] Decoder: parse `server_capabilities` from ConnectAck (same trailing-bytes approach)
- [x] Session: store negotiated capabilities in `AuthenticatedState`
- [x] Capability intersection logic in session Connect handler
- [x] Add `[deploy]` config section to `ServerConfig` with `drain_timeout_secs`, `shutdown_timeout_secs`, `going_away_reason`
- [x] Refactor `main.rs`: replace `watch::channel(false)` with `ServerLifecycle`; start HTTP server before `engine.init()`, set `initialized = true` after init completes
- [x] Refactor `main.rs` shutdown sequence: force-flush → Draining → wait → ShuttingDown
- [x] TCP accept loop: check `ServerLifecycle` state, stop accepting when `Draining`; don't accept until `initialized == true`
- [x] HTTP routes: `/health/live` → always 200; `/health/ready` → check `lifecycle.is_ready()`; `/health` → alias for `/health/ready`
- [x] HTTP routes: publish/query/info return 503 when `!lifecycle.initialized` (implemented as `require_initialized` axum middleware layer on `/v1/*`)
- [x] Inflight timeout acceleration on drain entry
- [x] Active connection counter: increment on accept, decrement on close; log progress during drain
- [x] Early exit from drain when connection count reaches 0
- [x] Config validation: `drain_timeout_secs >= 0`, `shutdown_timeout_secs >= 1`
- [x] Update `config/config.example.toml` with `[deploy]` section
- [x] Codec tests: Connect with and without capabilities field, round-trip, backward compat with v1 frames
- [x] Unit tests: `ServerLifecycle` state transitions, `is_ready()`, early exit, capability intersection
- [x] Integration test: startup sequence — `/health/live` responds before init, `/health/ready` transitions from 503 → 200
- [x] Integration test: v1 client (no caps) connects alongside capability-aware client, both authenticated successfully
- [x] Integration test: SIGTERM → drain → shutdown with connected clients (GoingAway not yet implemented — verified by state transitions and health endpoint responses)

#### Implementation notes (deviations from plan)

- Config section is `[deploy]` (not `[server]`), type is `DeployConfig`, to avoid collision with the existing top-level `ServerConfig` struct.
- The old `watch::channel(false)` shutdown signal was fully removed — all servers, background tasks, and connection handlers use `lifecycle.subscribe()` returning `watch::Receiver<ServerState>`. There is no dual-signal bridge.
- Business route init-gating uses an axum `route_layer` middleware (`require_initialized`) on `/v1/*` instead of per-handler checks, so new handlers added in future phases get gating automatically.
- Connection handler is drain-aware: subscribes to lifecycle, sends `ErrorCode::ServerDraining` on drain, rejects new `Subscribe`/`SubscribeAt`/`SubscribeFrom` frames during drain.
- `QueueEngine::force_flush()` delegates to new `StorageBackend::flush_all()` trait method (implemented on `FsStorage` and `MockStorage`), keeping flush logic in the storage layer.
- `ServerState` derives `PartialOrd`/`Ord` for comparison operators used in lifecycle checks.
- `TcpServer::new()` and `HttpServer::new()` no longer accept `watch::Receiver<bool>` — they take only `Arc<ServerLifecycle>`.
- `RetentionTask::run()` accepts `watch::Receiver<ServerState>` instead of `watch::Receiver<bool>`.

#### Files

| File | Change |
|---|---|
| `src/server/state.rs` | **New** — `ServerState`, `ServerLifecycle` |
| `src/server.rs` | Export `state` module |
| `src/codec/frame.rs` | `Capabilities` type, constants, `ServerDraining` error code, updated `Connect`/`ConnectAck` fields |
| `src/codec/encoder.rs` | Encode capabilities in Connect/ConnectAck |
| `src/codec/decoder.rs` | Backward-compatible capabilities parsing |
| `src/config.rs` | `DeployConfig` struct with `[deploy]` TOML section, drain/shutdown timeouts, validation |
| `src/main.rs` | Single lifecycle signal; start HTTP early; new shutdown sequence; all background tasks use lifecycle |
| `src/server/tcp.rs` | Lifecycle-driven accept loop; wait for initialized; stop on draining; connection counting |
| `src/server/tcp/connection.rs` | Lifecycle subscriber replaces shutdown_rx; drain-aware (ServerDraining error, subscribe rejection) |
| `src/server/tcp/session.rs` | Negotiated capabilities in `AuthenticatedState`; capability intersection in Connect handler |
| `src/server/http.rs` | Lifecycle-driven shutdown; `require_initialized` middleware on `/v1/*`; no shutdown_rx |
| `src/server/http/routes.rs` | `/health/live`, `/health/ready`, `require_initialized` middleware |
| `src/server/http/error.rs` | `ServerDraining` error code name |
| `src/namespace/manager.rs` | `set_inflight_timeout()` with `AtomicU64` for lock-free drain acceleration |
| `src/queue.rs` | `QueueEngine::force_flush()` trait method |
| `src/queue/engine.rs` | `force_flush()` implementation delegating to `storage.flush_all()` |
| `src/storage.rs` | `StorageBackend::flush_all()` trait method; MockStorage impl |
| `src/storage/fs.rs` | `FsStorage::flush_all()` implementation |
| `src/retention.rs` | `run()` accepts `watch::Receiver<ServerState>` |
| `config/config.example.toml` | Document `[deploy]` section |
| `tests/lifecycle_integration.rs` | **New** — 11 integration tests: capabilities, drain, health endpoints, init-gating |
| `tests/common/mod.rs` | Updated test helpers for new server APIs (no shutdown_rx, lifecycle-driven shutdown) |

---

### Phase 14.2 — `GoingAway` Frame & `ServerDraining` Error Code

**Goal:** Add a dedicated wire-level signal that tells clients "this is a planned shutdown, not a failure" with enough information to reconnect intelligently. This phase depends on 14.1 — capabilities must be negotiated before GoingAway can be safely sent.

#### New frame: `GoingAway` (0x10)

```
Direction: Server → Client only
MessageType: 0x10

Wire format:
  [u8  reason]              — why the server is going away
  [u32 deadline_ms]         — milliseconds until force disconnect
  [u16 message_length]      — length of human-readable message
  [u8* message]             — UTF-8 message string
```

**Reason codes — newtype `u8`, not an enum:**

```rust
/// Reason code for GoingAway frame. Newtype u8 for forward compatibility —
/// unknown reason values are accepted and logged, never rejected.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct GoingAwayReason(pub u8);

impl GoingAwayReason {
    /// Server is being upgraded to a new version.
    pub const UPGRADE: Self = Self(0x01);
    /// Server is being taken down for maintenance.
    pub const MAINTENANCE: Self = Self(0x02);
    /// Server is overloaded and shedding connections (reserved for future load-shedding).
    pub const OVERLOADED: Self = Self(0x03);
    /// Server is shutting down permanently.
    pub const SHUTDOWN: Self = Self(0x04);
}
```

> **Note:** `GoingAwayReason` is a newtype `u8`, not a Rust enum with `TryFrom`. This is intentional for forward compatibility: a future server version can send reason `0x05` and older clients just log "going away, reason=5" instead of hitting an unknown-variant error. The decoder never rejects an unknown reason — it always succeeds. Clients behave identically regardless of reason (ack inflight, disconnect, reconnect with jitter); the reason is informational for logging and operator dashboards.

> **Design decision:** `GoingAway` is NOT an `Error` frame. Error frames have `is_fatal()` semantics that close the connection immediately. GoingAway gives the client a grace period to finish work. Keeping them separate means error handling logic stays clean and GoingAway handling is its own concern.

#### `deadline_ms` semantics

- The server sends `deadline_ms` = remaining drain timeout at the moment the frame is sent
- This is a hint, not a contract. The server MAY force-close before the deadline if shutdown is urgent (e.g., cascading failure). Clients MUST handle early disconnection gracefully
- Clients SHOULD begin orderly disconnect (ack inflight, unsubscribe) immediately upon receiving GoingAway
- Clients SHOULD NOT start new subscriptions or publishes after receiving GoingAway
- Clients MAY use `deadline_ms` to schedule a reconnect timer (see backoff algorithm below)

**Reconnect backoff algorithm (for client SDK docs):**

```
reconnect_delay = deadline_ms + uniform_random(0, min(5000, deadline_ms / 2))
```

On connection failure after the initial attempt, exponential backoff:
```
retry_delay = min(base_delay * 2^attempt, 30_000)  // cap at 30s
```

> **Note:** The jitter formula is critical for thundering herd avoidance. If 5,000 clients all reconnect at exactly `deadline_ms`, the new server gets a connection spike that may exceed `max_connections`. The `min(5000, deadline_ms / 2)` jitter window scales with drain timeout — short drains get tight jitter, long drains get wide jitter.

#### New error code: `ServerDraining` (0x000D)

```rust
ErrorCode::ServerDraining = 0x000D,
```

- **Non-fatal.** Not in `is_fatal()`. The connection is not closed by this error alone.
- Returned when a client attempts `Subscribe`, `SubscribeAt`, `SubscribeFrom`, or `Connect` during drain mode.
- Publishes are always accepted during drain (in-flight work must complete).
- Tells the client "your request was rejected because the server is draining, not because the request was invalid."
- A well-behaved client that processes `GoingAway` immediately will never see `ServerDraining`. A slow or buggy client that ignores `GoingAway` and tries to subscribe will get this.

#### Session state: `Draining`

When the connection handler detects `Draining` state (via lifecycle watcher):

1. Check per-session capabilities:
   - `CAP_GOING_AWAY` set → send `GoingAway { reason, deadline_ms: remaining_drain_ms, message }`
   - `CAP_GOING_AWAY` not set → send `Error { code: InternalError, message: "server shutting down" }` (current behavior, no regression)
2. Transition session to a new internal state: `Draining` (between `Authenticated` and disconnected)
3. In `Draining` state:
   - `Publish` → allowed (in-flight work completes)
   - `Subscribe` / `SubscribeAt` / `SubscribeFrom` → `ServerDraining` error
   - `Ack` → allowed (clients should ack inflight messages)
   - `Unsubscribe` → allowed (clients should clean up)
   - `Ping` / `Pong` → allowed (keepalive continues during drain)
4. Delivery tasks continue running — messages already queued are still delivered
5. When client disconnects (voluntarily or force-closed at drain timeout), normal cleanup runs

#### Work items

- [x] Add `GoingAway` variant to `Frame` enum with `reason: GoingAwayReason`, `deadline_ms: u32`, `message: String`
- [x] Add `GoingAwayReason` newtype struct with const values to `src/codec/frame.rs`
- [x] Add `MessageType::GoingAway = 0x10` to `MessageType` enum
- [x] Implement encoder for `GoingAway` frame
- [x] Implement decoder for `GoingAway` frame — accept any `u8` reason value (forward compat)
- [x] Add `ErrorCode::ServerDraining = 0x000D` — non-fatal (not in `is_fatal()`) *(already existed from 14.1)*
- [x] Add `Session::Draining(AuthenticatedState)` to session state machine
- [x] Implement frame dispatch rules for `Draining` state (allow Ack/Unsub/Ping/Pong/Publish, reject Subscribe variants)
- [x] Connection handler: on lifecycle `Draining` transition, check capabilities, send GoingAway or Error, transition session to `Draining`
- [x] Connection-level drain deadline: force-close safety net after `deadline_ms + 1s` if `ShuttingDown` never arrives
- [x] Connection-level `lifecycle.is_draining()` guard for subscribe frames — closes race window between frame read and session drain transition
- [x] Config validation: reject unknown `going_away_reason` values at startup
- [x] `GoingAwayReason::from_config()` logs warning for unknown reason strings
- [x] Codec round-trip tests for `GoingAway` frame (all reason codes, unknown reason code, empty message, max-length message)
- [x] Encoder edge-case tests: `deadline_ms = 0`, `deadline_ms = u32::MAX`
- [x] Session state machine tests: 7 tests for `Draining` state (rejects all 3 subscribe variants, allows publish, allows unsubscribe, no-op on pre-auth, preserves capabilities)
- [x] Integration test: capability-aware client receives `GoingAway`, acks inflight, disconnects; server proceeds to shutdown
- [x] Integration test: client without capabilities receives `Error { ServerDraining }` on drain — no regression
- [x] Integration test: client sends `Subscribe` during drain (after GoingAway) → receives `ServerDraining` error
- [x] Integration test: publish accepted during drain for capable client (PublishAck returned after GoingAway)

#### Files

| File | Change |
|---|---|
| `src/codec/frame.rs` | `GoingAwayReason` newtype with `from_config()`, `MessageType::GoingAway = 0x10`, `Frame::GoingAway` variant |
| `src/codec/encoder.rs` | Encode `GoingAway`: reason (u8) + deadline_ms (u32) + message (string) |
| `src/codec/decoder.rs` | Decode `GoingAway` with forward-compat reason parsing; 4 roundtrip tests + proptest coverage |
| `src/server/tcp/session.rs` | `Session::Draining(AuthenticatedState)` variant, `transition_to_draining()`, `capabilities()` accessor, subscribe rejection in `Draining` state, `GoingAway` in server-only frame rejection, 7 new unit tests |
| `src/server/tcp/connection.rs` | `DeployConfig` on `ConnectionConfig`, capabilities-aware GoingAway vs Error, connection-level drain deadline (safety net), lifecycle guard for subscribe race window |
| `src/server/tcp.rs` | `TcpServer` accepts `DeployConfig`, threads to `ConnectionConfig` |
| `src/main.rs` | Pass `config.deploy` to `TcpServer::new()` |
| `src/config.rs` | Config validation: reject unknown `going_away_reason` values |
| `tests/lifecycle_integration.rs` | 4 new integration tests: GoingAway for capable client, Error for non-capable, subscribe rejected during drain, publish accepted during drain |

#### Implementation notes (deviations from plan)

- `Session::Draining(AuthenticatedState)` added as a proper enum variant. The session itself rejects `Subscribe`/`SubscribeAt`/`SubscribeFrom` with `ServerDraining` when in `Draining` state. A belt-and-suspenders `lifecycle.is_draining()` guard at the connection level closes the race window where a subscribe frame is read from the TCP buffer in the same `select!` iteration as the drain transition (before the session has been transitioned).
- `ConnectionConfig` holds `deploy: DeployConfig` (whole struct, not individual fields) for forward-compatibility as `DeployConfig` gains fields in future phases.
- Connection-level drain deadline set to `deadline_ms + 1 second` buffer, so the lifecycle's `ShuttingDown` transition wins in the normal case. The connection deadline only fires as a safety net if `ShuttingDown` never arrives.
- `GoingAwayReason::from_config()` logs a warning for unknown strings. Config validation also rejects unknown `going_away_reason` values at startup, so the runtime warning is a defense-in-depth backstop.
- Capable clients (CAP_GOING_AWAY) do NOT have their connection closed after receiving GoingAway — they stay connected and can finish inflight work. The `ShuttingDown` transition (or the drain deadline safety net) force-closes remaining connections. Non-capable clients get `Error { ServerDraining }` and immediately disconnect.
- `ErrorCode::ServerDraining` was already implemented in 14.1, not `InternalError` — the plan originally said non-capable clients get `Error { InternalError }` but they actually get `ServerDraining` which is more informative.

---

### Phase 14.3 — `Reconnect` Frame & Offset Resume

**Goal:** Let clients restore their full subscription state in a single round-trip after a deploy, instead of manually re-subscribing to each topic with `SubscribeAt`. This is an optimization — `Connect + SubscribeAt` always works and remains the primary reconnect path.

> **Sequencing note:** This phase ships after 14.1 + 14.2. The core deploy story (drain + GoingAway + health probes + capabilities) is fully functional without Reconnect. Clients can deploy, receive GoingAway, and reconnect using the existing `Connect + SubscribeAt` flow. This phase adds a protocol optimization that becomes valuable as client subscription counts and network latencies grow. It is NOT required for safe deploys.

#### Problem

After receiving `GoingAway`, a client with N topic subscriptions must:
1. Track offsets for all N topics locally
2. Reconnect with `Connect`
3. Send N `SubscribeAt` frames, each requiring a `SubscribeAck` response
4. Wait for all N acks before delivery resumes

That's `1 + 2N` round-trips. For most clients (1-5 subscriptions, same-DC RTT <1ms), this is imperceptible. For clients with many subscriptions in high-latency environments (50 topics, 50ms cross-region RTT), this is 5 seconds of dead time. `Reconnect` collapses this to 2 round-trips regardless of subscription count.

#### New frame: `Reconnect` (0x11)

```
Direction: Client → Server
MessageType: 0x11

Wire format:
  [u8  version]                — protocol version (must be 1)
  [u16 namespace_length]       — namespace string length
  [u8* namespace]              — namespace UTF-8 string
  [u16 api_key_length]         — API key string length
  [u8* api_key]                — API key UTF-8 string
  [u32 capabilities]           — client capabilities (same as Connect)
  [u16 subscription_count]     — number of subscriptions to restore
  [subscription_entry]*        — repeated subscription_count times

subscription_entry:
  [u16 topic_length]           — topic string length
  [u8* topic]                  — topic UTF-8 string
  [u16 group_length]           — group string length (0 = individual subscriber)
  [u8* group]                  — group UTF-8 string (absent if length=0)
  [u64 last_acked_offset]      — the last offset the client successfully acked
```

> **Note:** `last_acked_offset` is the client's durable checkpoint. The server resumes delivery from `last_acked_offset + 1`. If the offset is below the oldest retained segment, the server clamps to the oldest available and reports `offset_clamped` in the per-subscription result.

> **Requirement:** `Reconnect` requires the `CAP_RECONNECT` capability to have been offered by the server during a prior connection (or discovered out of band). If a client sends `Reconnect` to a server that doesn't support it, the server treats it as an unknown pre-auth frame and returns `Error { InvalidFrame }`.

#### New frame: `ReconnectAck` (0x12)

```
Direction: Server → Client
MessageType: 0x12

Wire format:
  [u8  status]                 — 0x00 = success, 0x01 = partial, 0x02 = auth failed
  [u16 message_length]         — message string length
  [u8* message]                — UTF-8 status message
  [u32 server_capabilities]    — negotiated capabilities (same as ConnectAck)
  [u16 result_count]           — number of subscription results
  [subscription_result]*       — repeated result_count times

subscription_result:
  [u16 topic_length]           — topic string length
  [u8* topic]                  — topic UTF-8 string
  [u16 group_length]           — group string length
  [u8* group]                  — group UTF-8 string
  [u8  sub_status]             — 0x00 = restored, 0x01 = offset_clamped,
                                 0x02 = topic_not_found, 0x03 = permission_denied,
                                 0x04 = group_full
  [u64 actual_start_offset]    — where delivery will actually start
```

**Status semantics:**
- `success` (0x00): All subscriptions restored at requested offsets. OffsetClamped counts as success (it's a warning, not a failure).
- `partial` (0x01): Some subscriptions had hard failures (TopicNotFound, PermissionDenied, GroupFull). **Atomic rollback:** all successful subscriptions are unsubscribed — no active subscriptions exist. Connection IS established — client should inspect per-subscription results and use `Connect + SubscribeAt` for individual control.
- `auth_failed` (0x02): API key invalid, namespace mismatch, or connection rejected. Connection closed. Same as `ConnectAck { success: false }`.

> **Note:** The atomic rollback ensures the client never ends up with a partial set of subscriptions it didn't ask for. If 48 of 50 subscriptions succeed but 2 fail, the client gets `partial` with all 50 results, and zero active subscriptions. It can then fall back to `Connect + SubscribeAt` to subscribe to the 48 that worked individually. This is safer than the original plan's "keep the 48" approach — it prevents orphaned forwarders and gives the client full control.

#### Reconnect vs Connect + N x SubscribeAt

| | Connect + SubscribeAt | Reconnect |
|---|---|---|
| Round-trips | 1 + 2N | 2 (Reconnect + ReconnectAck) |
| Client complexity | Must track offsets + re-sub loop | Send one frame, check one response |
| Atomicity | Race: messages can arrive between sub #3 and sub #50 | All-or-nothing: all restored before delivery starts, or all rolled back on partial failure |
| Backward compatible | Yes (existing frames) | Requires server and client to support 0x11/0x12 |
| When to use | Always works, good for 1-5 subscriptions | High subscription count or high-latency networks |

> **Note:** `Connect + SubscribeAt` remains fully supported and is the recommended default for simple clients. `Reconnect` is an optimization for clients that implement it. The server never requires `Reconnect` — it's always optional.

#### Server-side handling

1. Validate version (must be 1)
2. Authenticate via API key (same as `Connect`)
3. **Validate namespace** — frame's `namespace` must match the namespace resolved from the API key. Mismatch → `AuthFailed` with "namespace mismatch" message.
4. Negotiate capabilities (same as `Connect` — `agreed = client_caps & SERVER_CAPABILITIES`)
5. Call `manager.connect()` for namespace connection limit
6. Validate subscription count <= 10,000 (prevent abuse)
7. **Batch subscribe (atomic)** — for each subscription entry:
   a. Check Subscribe permission (once, applies to all entries)
   b. Call `manager.subscribe()` with `SubscribeStart::Offset(last_acked_offset + 1)`
   c. Record per-subscription result and hold receiver (don't spawn forwarder yet)
8. **Commit or rollback:**
   - If no hard failures → spawn forwarders for all receivers, insert into session subscriptions
   - If any hard failure (TopicNotFound, PermissionDenied, GroupFull) → call `manager.unsubscribe()` for all successful subscriptions, drop receivers, no forwarders spawned
   - OffsetClamped is a warning, NOT a hard failure — does not trigger rollback
9. Transition session to `Authenticated` state
10. Build `ReconnectAck` with aggregated results and negotiated capabilities

> **Important:** All subscriptions are committed (forwarders spawned) BEFORE the `ReconnectAck` is sent and BEFORE any `Deliver` frames are sent. This prevents the race condition where Connect + SubscribeAt would start delivering to topic A while topic B hasn't been subscribed yet. In the rollback case, no forwarders are spawned at all.

#### Work items

- [x] Add `Reconnect` variant to `Frame` enum
- [x] Add `ReconnectAck` variant to `Frame` enum with per-subscription results
- [x] Add `MessageType::Reconnect = 0x11`, `MessageType::ReconnectAck = 0x12`
- [x] Implement encoder/decoder for `Reconnect` frame
- [x] Implement encoder/decoder for `ReconnectAck` frame
- [x] Session state machine: handle `Reconnect` in `PreAuth` state (like `Connect` but with subscription restoration and capability negotiation)
- [x] Server-side reconnect logic: auth → negotiate caps → connect → batch subscribe → build ack
- [x] Validate: max subscription count per `Reconnect` frame (cap at 10,000 to prevent abuse)
- [x] Handle offset clamping: if `last_acked_offset + 1` < oldest available, clamp and report `offset_clamped`
- [x] Codec round-trip tests for `Reconnect` and `ReconnectAck` frames
- [x] Session tests: `Reconnect` with valid/invalid auth, partial failures, offset clamping, capability negotiation
- [x] Integration test: subscribe to topic → disconnect → reconnect with `Reconnect` frame → verify delivery resumes at correct offset
- [x] Integration test: `Reconnect` with partial failure (no subscribe permission) → verify Partial status, no active subscriptions (rollback)
- [x] Integration test: `Reconnect` with bad auth → verify AuthFailed and connection close

#### Files

| File | Change |
|---|---|
| `src/codec/frame.rs` | `Reconnect`, `ReconnectAck` frame variants, `MessageType` additions, `ReconnectStatus`, `SubscriptionStatus`, `SubscriptionEntry`, `SubscriptionResult` types |
| `src/codec/encoder.rs` | Encode `Reconnect`, `ReconnectAck` with nested subscription lists |
| `src/codec/decoder.rs` | Decode `Reconnect`, `ReconnectAck`; 5 roundtrip tests; proptest coverage |
| `src/server/tcp/session.rs` | `process_reconnect()` with namespace validation, atomic batch subscribe + rollback, capability negotiation; `ReconnectAck` in server-only rejection; `Reconnect` rejected in Authenticated state; 9 new unit tests |
| `src/server/tcp/connection.rs` | Drain guard extended to reject `Reconnect` alongside `Subscribe`/`SubscribeAt`/`SubscribeFrom` |
| `tests/tcp_integration.rs` | 3 new integration tests: offset resume, partial failure, auth failure |
| `CLAUDE.md` | Documented Reconnect frame semantics, atomic batch subscribe, namespace validation |
| `docs/protocol.md` | *(deferred to Phase 14.6)* Document `Reconnect` / `ReconnectAck` frames |

#### Implementation notes (deviations from plan)

- Batch subscribe is implemented directly in `Session::process_reconnect()` rather than as a separate `NamespaceManager::batch_subscribe()` method. This avoids a trait change and keeps the logic co-located with authentication and capability negotiation.
- **Atomic batch subscribe:** All subscriptions are attempted first without spawning forwarders. If any hard failure occurs (TopicNotFound, PermissionDenied, GroupFull), all successful subscriptions are rolled back via `manager.unsubscribe()`. The client gets `Partial` status with per-subscription details but NO active subscriptions. OffsetClamped is a warning, not a hard failure — it doesn't trigger rollback.
- `ReconnectStatus::AuthFailed` is also returned when `manager.connect()` rejects the connection (e.g., namespace connection limit reached). The message field distinguishes: "authentication failed" vs "connection rejected: {error}" so clients can tell the difference.
- **Namespace validation:** The `namespace` field in the `Reconnect` frame is validated against the authenticated namespace from the API key. Mismatches return `ReconnectStatus::AuthFailed` with "namespace mismatch" message. This catches client bugs where a reconnect is sent to the wrong server.
- `ReconnectAck` is added to the server-only frame rejection list in `process_frame`, alongside `ConnectAck`, `GoingAway`, etc.
- `Reconnect` in `Authenticated` or `Draining` state returns `InvalidFrame` ("already authenticated, Reconnect only valid before auth") via `process_authenticated`.
- The connection handler's lifecycle drain guard also rejects `Reconnect` frames during drain, alongside `Subscribe`/`SubscribeAt`/`SubscribeFrom`.
- `ReconnectStatus`, `SubscriptionStatus`, `SubscriptionEntry`, and `SubscriptionResult` types are defined in `frame.rs` as public types.
- Proptest coverage added for both `Reconnect` and `ReconnectAck` frames in the `arb_frame()` strategy.
- Integration tests added to `tests/tcp_integration.rs`: offset resume, partial failure with permission denied, auth failure.

---

### Phase 14.4 — Graceful Deploy Integration Test Suite

**Goal:** End-to-end tests that validate the complete deploy lifecycle with concurrent publishers and consumers, simulating real production scenarios.

#### Test scenarios

```
14.4.1: Single-instance deploy — basic
  - Start server
  - Connect 10 subscribers (with CAP_GOING_AWAY) across 3 topics
  - Connect 5 publishers, publish 1000 messages
  - Send SIGTERM
  - Assert: all capability-aware clients receive GoingAway
  - Assert: clients that ack inflight and disconnect are not force-closed
  - Assert: /health/ready returns 503 during drain
  - Assert: server exits 0

14.4.2: Offset continuity across restart
  - Start server
  - Subscribe to topic, consume 500 of 1000 messages, ack all
  - Send SIGTERM, wait for shutdown
  - Start new server instance (same data dir)
  - Connect + SubscribeAt with last acked offset
  - Assert: delivery resumes at offset 501
  - Assert: no duplicate messages, no gap

14.4.3: Consumer group continuity
  - Start server with 3-member consumer group on topic
  - Publish 1000 messages, consume ~500 across group
  - Send SIGTERM
  - Start new server
  - Reconnect all 3 group members with their last acked offsets (via SubscribeAt)
  - Assert: no message delivered twice across the group
  - Assert: all 1000 messages eventually delivered exactly once

14.4.4: Drain timeout with slow client
  - Start server with drain_timeout_secs = 5
  - Connect client that ignores GoingAway (never disconnects)
  - Send SIGTERM
  - Assert: GoingAway sent immediately
  - Assert: after 5 seconds, client is force-closed
  - Assert: server exits 0

14.4.5: Publish during drain
  - Start server
  - Connect publisher and subscriber (both capability-aware)
  - Send SIGTERM
  - Publisher sends 100 more messages during drain
  - Assert: all 100 messages delivered to subscriber
  - Assert: PublishAck received for all 100
  - Assert: messages are persisted (available after restart)

14.4.6: Legacy client (no capabilities) during deploy
  - Start server
  - Connect legacy client (no capabilities in Connect frame)
  - Connect capability-aware client (with CAP_GOING_AWAY)
  - Send SIGTERM
  - Assert: legacy client receives Error { InternalError, "server shutting down" }
  - Assert: capability-aware client receives GoingAway
  - Assert: both eventually disconnect, server shuts down cleanly

14.4.7: Reconnect frame — full restore (14.3 complete — basic coverage in tcp_integration.rs)
  - Start server with 3 topics
  - Connect with CAP_RECONNECT, subscribe to all 3, consume some messages, ack
  - Send SIGTERM, wait for shutdown
  - Start new server
  - Send Reconnect with all 3 subscriptions and last acked offsets
  - Assert: ReconnectAck status = success
  - Assert: delivery resumes at correct offset for each topic
  NOTE: Single-topic offset resume is covered by `reconnect_resumes_delivery_at_correct_offset`
  in tcp_integration.rs. This test extends to multi-topic + full deploy cycle.

14.4.8: Reconnect frame — partial failure (14.3 complete — basic coverage in tcp_integration.rs)
  - Start server with 3 topics
  - Subscribe to all 3, consume some messages
  - Stop server, delete topic B's data
  - Start new server
  - Send Reconnect with all 3 subscriptions
  - Assert: ReconnectAck status = partial
  - Assert: topic B = topic_not_found, topic A and C = restored
  - Assert: NO active subscriptions (atomic rollback) — no deliveries arrive
  - Assert: client can recover via Connect + SubscribeAt for topics A and C
  NOTE: Permission-based partial failure is covered by `reconnect_partial_no_subscribe_permission`
  in tcp_integration.rs. This test covers data-loss partial failure + manual recovery path.

14.4.9: Thundering herd avoidance (stress, #[ignore])
  - Start server
  - Connect 1000 clients (all capability-aware)
  - Send SIGTERM
  - Start new server
  - All 1000 clients reconnect with staggered jitter
  - Assert: new server handles connection ramp without reject spike
  - Assert: all clients eventually connected and receiving

14.4.10: Storage durability through drain
  - Start server, publish 10000 messages with batch_flush_interval_ms = 60000
    (very long flush interval — data sits in buffer)
  - Send SIGTERM
  - Assert: force-flush runs at drain entry
  - Start new server (same data dir)
  - Assert: all 10000 messages available (none lost from unflushed buffer)
```

> **Note:** Tests 14.4.7, 14.4.8 depend on phase 14.3 (Reconnect frame) — now complete. No feature-gating needed. Basic Reconnect coverage already exists in `tests/tcp_integration.rs` (offset resume, partial failure, auth failure). Tests 14.4.7–8 extend this to multi-topic + full deploy lifecycle (SIGTERM → restart → reconnect). Test 14.4.9 is a stress test — tag with `#[ignore]` and run in nightly/release CI. The rest should run in < 30 seconds total.

#### Work items

- [x] Create `tests/deploy_integration.rs`
- [x] Implement test helper: `DeployTestServer` struct combining TCP + HTTP + FsStorage, with `graceful_drain()` and `shutdown()` methods
- [x] Implement test helper: `start_deploy_test_server(data_dir, deploy_config, engine_config)` — FsStorage-backed server factory
- [x] Implement test helper: `TestClient::connect_and_auth_with_caps(addr, ns, key, caps)` — capabilities-aware auth
- [x] Implement local helpers: `publish_messages()`, `subscribe_at_and_consume()`, `collect_delivers()`
- [x] Tests 14.4.1 – 14.4.6, 14.4.10: standard integration tests (core deploy, no Reconnect dependency)
- [x] Tests 14.4.7 – 14.4.8: Reconnect frame tests (14.3 complete, no feature gate needed)
- [x] Test 14.4.9: tagged `#[ignore]`, stress test (1000 clients)
- [x] Additional test: `data_loss_recovery_across_restart` — topic data deleted from disk, surviving topics verified
- [ ] CI configuration: run non-ignored tests in standard pipeline, ignored tests in nightly/release pipeline

#### Files

| File | Change |
|---|---|
| `tests/deploy_integration.rs` | **New** — 11 test functions (10 active + 1 `#[ignore]` stress test) + local helper functions |
| `tests/common/mod.rs` | `DeployTestServer` struct, `start_deploy_test_server()`, `TestClient::connect_and_auth_with_caps()`, new imports (`FsStorage`, `StorageBackend`, `QueueEngine`, `NamespaceManager`, `Capabilities`) |

#### Implementation notes (deviations from plan)

- **Test infrastructure uses `DeployTestServer`** instead of the planned `start_server_with_config()` + `mock_client()`. `DeployTestServer` bundles TCP + HTTP + FsStorage + `ServerLifecycle` in one struct with `graceful_drain()` and `shutdown()` methods that replicate the `main.rs` shutdown sequence. This is more ergonomic and ensures consistent drain behavior across all tests.
- **`consume_and_ack()` was removed** during development. `Frame::Subscribe` starts at head (latest offset), so subscribing after publishing yields no messages. All tests use `subscribe_at_and_consume()` with explicit offset 0 instead. This matches the real-world reconnect pattern where clients track their own offsets.
- **14.4.3 (consumer group continuity) simplified**: The plan called for 3 consumer group members with round-robin delivery. Since consumer group state is purely in-memory (not persisted), the test was simplified to use 3 individual subscribers consuming contiguous offset ranges, then a single subscriber resuming from the max offset after restart. This proves the same invariant (all messages delivered, no gaps) with deterministic behavior and no JoinSet complexity.
- **14.4.4 (drain timeout) uses 3s** instead of 5s to keep test runtime short. Assertion window is 3–6s.
- **14.4.6 (legacy client)**: Legacy client receives `ErrorCode::ServerDraining` (not `InternalError` as the plan originally stated). This was already the behavior from Phase 14.2.
- **14.4.8 (Reconnect partial failure) uses permission denial** instead of deleted topic data. The engine auto-creates topics on subscribe, so deleting topic data from disk does not produce `TopicNotFound` — the engine creates an empty topic. The test instead starts a server with publish-only permissions (no Subscribe), which triggers `SubscriptionStatus::PermissionDenied` and `ReconnectStatus::Partial` with atomic rollback. A separate `data_loss_recovery_across_restart` test covers the plan's intended data-loss scenario: topic "beta" deleted from disk, restart, verify alpha/gamma survive and beta is empty.
- **Tests run in parallel** (`--test-threads=4`). Each test uses its own `TempDir` and binds to port 0, so there are no resource conflicts. Suite completes in ~120s (vs ~270s serial).

---

### Phase 14.5 — Operational Tooling

**Goal:** Give operators the tools to trigger and monitor graceful deploys without relying solely on SIGTERM.

#### Admin authentication

Admin endpoints use a dedicated `admin_api_key` in the `[deploy]` config section. This avoids modifying the existing `Permission` enum and auth system, which are namespace-scoped. Admin operations are server-global and don't belong to any namespace.

```toml
[deploy]
# API key for admin endpoints (/admin/*). Must be >= 24 characters.
# If not set, admin endpoints return 403. Env: FAFB_ADMIN_API_KEY
admin_api_key = "your-admin-key-here-minimum-24-chars"
```

> **Note:** This is deliberately simple. Admin auth is a single key comparison — no namespace scoping, no permission matrix. The admin key is checked via `Authorization: Bearer <key>` header, same as namespace keys. If the key is not configured, admin endpoints return 403 with a message directing operators to set it. This avoids touching `Permission`, `ApiKeyStore`, or any namespace config.

#### HTTP drain endpoint

```
POST /admin/drain
  Auth: admin_api_key via Bearer token
  Body: { "reason": "upgrade", "timeout_secs": 30 }  (optional, overrides config)
  Response (200): { "status": "draining", "active_connections": 4523,
                    "deadline": "2026-03-21T15:30:00Z" }
  Response (409): { "status": "already_draining", ... }  (idempotent — returns current state)
  Response (403): { "error": "admin API key not configured or invalid" }

  Triggers the same drain sequence as SIGTERM, but via HTTP.
  Allows orchestration tools to drain the server before sending SIGTERM.
```

> **Note:** This is useful for orchestration that goes beyond "send SIGTERM":
> - A deploy script can `POST /admin/drain`, poll `/health/ready` until 503, then `kill` the process
> - A canary deploy can drain the old instance before shifting traffic
> - An operator can trigger drain from a dashboard without SSH access

#### SIGUSR1 — drain without shutdown (stretch goal)

```
SIGUSR1 → enter drain mode but do NOT proceed to ShuttingDown
  - Server stops accepting new connections
  - Sends GoingAway to all clients
  - /health/ready returns 503
  - But: does NOT force-close after drain_timeout, does NOT call engine.shutdown()
  - Server stays running in drained state

POST /admin/drain/cancel → return to Running state
  - Resume accepting connections
  - /health/ready returns 200
```

> **Note:** Useful for operators who want to drain a server for debugging or maintenance without killing it. SIGTERM = drain + shutdown. SIGUSR1 = drain only. Low effort, high operator value. If time is short, defer this to a follow-up — the core deploy story doesn't need it.

#### Drain status endpoint

```
GET /admin/drain/status
  Auth: admin_api_key
  Response: {
    "state": "running"|"draining"|"shutting_down",
    "active_connections": 4523,
    "drain_started_at": "2026-03-21T15:29:30Z",  // null if not draining
    "drain_deadline": "2026-03-21T15:30:00Z",     // null if not draining
    "connections_drained": 477                      // since drain started
  }
```

> **Note:** The `inflight_messages` field from the original plan is removed — aggregating inflight across all namespaces/topics on every status poll is expensive. Use the existing `fafb_inflight_messages` Prometheus gauge instead. The status endpoint focuses on connection-level drain progress, which is cheap to compute from `ServerLifecycle` atomics.

#### Metrics additions

```
fafb_server_state                gauge    — 0=running, 1=draining, 2=shutting_down
fafb_drain_duration_seconds      histogram — time from drain start to all connections closed
fafb_going_away_sent_total       counter  — GoingAway frames sent
fafb_reconnect_total             counter  — Reconnect frames processed (labels: status=success|partial|auth_failed)
fafb_reconnect_subscriptions     histogram — subscriptions per Reconnect frame
```

#### Work items

- [x] Add `admin_api_key: Option<String>` to `[deploy]` config section (not `[server]` — see implementation notes)
- [x] Config validation: if set, must be >= 24 characters
- [x] Admin auth middleware: check `Authorization: Bearer` against `admin_api_key`; return 403 if not configured or invalid
- [x] `POST /admin/drain` endpoint — trigger drain via HTTP, with optional body overrides
- [x] `GET /admin/drain/status` endpoint — return drain progress from `ServerLifecycle` atomics
- [x] Add drain-related Prometheus metrics
- [ ] (Deferred) SIGUSR1 handler for drain-without-shutdown — low value now that HTTP drain exists
- [ ] (Deferred) `POST /admin/drain/cancel` to return to Running state — breaks forward-only lifecycle invariant, needs own phase
- [x] Integration test: HTTP-triggered drain → GoingAway → shutdown
- [x] Integration test: admin endpoint returns 403 when key not configured
- [x] Update `config/config.example.toml` with `admin_api_key`

#### Files

| File | Change |
|---|---|
| `src/server/http/routes.rs` | `/admin/drain`, `/admin/drain/status` endpoints, `require_admin_auth` middleware |
| `src/server/http.rs` | `AppState` gains `admin_api_key`, `drain_trigger`, `deploy_config`; `DrainRequest`/`DrainResponse` types; router adds `/admin` nest; `HttpServer::new(config, state)` simplified to 2-arg constructor |
| `src/server/state.rs` | `begin_drain()`, `run_drain_wait()`, `drain_started_wall`, `drain_deadline_wall`, `connections_drained()`, `fafb_server_state` gauge |
| `src/types.rs` | `millis_to_rfc3339()` public utility alongside `now_millis()` |
| `src/config.rs` | `admin_api_key: Option<String>` in `DeployConfig`, validation, env override, 5 new unit tests |
| `src/main.rs` | Drain channel creation, `drain_rx` select branch, `begin_drain()` call, server state gauge init; constructs `AppState` directly |
| `src/server/tcp/connection.rs` | `fafb_going_away_sent_total` counter |
| `src/server/tcp/session.rs` | `fafb_reconnect_total` counter, `fafb_reconnect_subscriptions` histogram |
| `config/config.example.toml` | Document `admin_api_key` |
| `tests/common/mod.rs` | `start_admin_test_server()` helper, `graceful_drain()` uses `begin_drain()` + `run_drain_wait()`, `HttpServer::new()` calls updated to `AppState` struct |
| `tests/deploy_integration.rs` | 6 new admin integration tests, `handle_drain_request()` takes individual components (no fake `DeployTestServer`) |
| `CLAUDE.md` | Admin endpoints, metrics, `begin_drain()`, `run_drain_wait()`, config documentation |

#### Implementation notes (deviations from plan)

- `admin_api_key` placed in `[deploy]` config section (not `[server]` — no `[server]` section exists). `DeployConfig` already holds deployment-related config.
- Admin auth uses `subtle::ConstantTimeEq` (already a dependency from `src/auth/apikey.rs`) for constant-time key comparison.
- Admin routes live at `/admin/*` (outside `/v1/*`) so they bypass the `require_initialized` middleware and work during startup and drain.
- HTTP-triggered drain uses `mpsc::channel<DrainRequest>` with `oneshot` response. The HTTP handler sends a `DrainRequest`, main.rs receives it, runs the drain sequence, and sends `DrainResponse` back. This keeps drain orchestration in one place (main.rs).
- `ServerLifecycle::begin_drain(timeout_secs)` introduced alongside `transition_to()`. `begin_drain()` records wall-clock timestamps and sets the `fafb_server_state` gauge. `transition_to()` still works for ShuttingDown and also sets the gauge.
- `ServerLifecycle::run_drain_wait(timeout)` extracts the drain-wait loop + drain duration metric + `ShuttingDown` transition into a shared method. Used by `main.rs`, `DeployTestServer::graceful_drain()`, and test `handle_drain_request()` — eliminates triple duplication.
- `millis_to_rfc3339()` placed in `crate::types` (alongside `now_millis()`) as a public utility. Uses Howard Hinnant's `civil_from_days` algorithm — zero external deps.
- `HttpServer::new(config, state)` refactored from 9 positional arguments to 2 (`HttpConfig` + `AppState`). Callers construct `AppState` with named fields.
- `DeployTestServer::graceful_drain()` uses `begin_drain()` + `run_drain_wait()` instead of manual `transition_to(Draining)` + inline wait loop. Wall-clock timestamps now consistently recorded in all drain paths.
- Test `handle_drain_request()` takes `Arc<ServerLifecycle>`, `Arc<Engine>`, `Arc<Manager>`, `timeout_secs` directly — no fake `DeployTestServer` with dummy fields.
- 5 config unit tests: admin_api_key None passes, too-short fails, valid passes, env override sets key, empty env override clears.
- 6 admin integration tests: auth 403 (no key), auth 403 (wrong key), drain status shows running, drain triggers shutdown with GoingAway, 409 already draining, status reflects connection progress.

---

### Phase 14.6 — Documentation

**Goal:** Comprehensive documentation for operators deploying the server and developers building client SDKs. This phase produces documentation only — no code changes.

#### Documents to create/update

- [x] **`docs/protocol.md`** — update with:
  - Capability negotiation in `Connect`/`ConnectAck` (wire format, backward compat rules)
  - `GoingAway` frame (0x10): wire format, reason codes (with note: unknown reasons MUST be accepted), `deadline_ms` semantics
  - `Reconnect` frame (0x11): wire format, subscription entry format, `last_acked_offset` semantics
  - `ReconnectAck` frame (0x12): wire format, per-subscription status codes, partial success semantics
  - `ServerDraining` error code (0x000D): non-fatal, when returned, client behavior
  - Capability bits table with assigned values
  - Client offset tracking requirement: "Clients MUST track the `offset` field from every `Deliver` frame they acknowledge. This offset is required for resuming consumption after a disconnect via `SubscribeAt` or `Reconnect`."
  - Example byte sequences for new frames

- [x] **`docs/deploy.md`** (new) — Operator deploy guide:
  - Single-instance deploy procedure (SIGTERM flow, what happens at each step)
  - Rolling deploy procedure (multi-instance behind LB, readiness-based traffic shifting)
  - Config reference for `[deploy]` section (`drain_timeout_secs`, `shutdown_timeout_secs`, `going_away_reason`, `admin_api_key`)
  - Health check integration (k8s probe config examples, ECS health check config, HAProxy config, nginx upstream config)
  - HTTP drain API usage (`POST /admin/drain` workflow for scripted deploys)
  - Monitoring during deploy (which metrics to watch: `fafb_server_state`, `fafb_connections_active`, `fafb_going_away_sent_total`; what alerts to set)
  - Troubleshooting: "clients not reconnecting", "drain timeout too short", "thundering herd after restart", "health probe failing during startup"
  - Upgrade checklist: pre-deploy verification, during-deploy monitoring, post-deploy validation

- [x] **`docs/client_sdk_deploy.md`** (new) — Client SDK developer guide:
  - Capability advertisement: how to send capabilities in `Connect`, how to read `server_capabilities` from `ConnectAck`
  - GoingAway handling: receive → ack inflight → unsubscribe → disconnect → reconnect with jitter
  - Offset tracking requirement: MUST track highest acked offset per topic per subscription
  - Reconnect strategies:
    - **Simple (recommended for most clients):** `Connect` + `SubscribeAt` per topic — works with all server versions
    - **Optimized (high subscription count):** `Reconnect` frame — single round-trip, requires `CAP_RECONNECT`
  - Reconnect backoff algorithm: `deadline_ms + uniform_random(0, min(5000, deadline_ms / 2))`, exponential backoff `min(base * 2^attempt, 30s)` on failure
  - Error code handling: `ServerDraining` → stop new requests, finish inflight, disconnect; `InternalError` with "server shutting down" → legacy server, reconnect with standard backoff
  - Example: full reconnect flow in pseudocode (language-agnostic)
  - Anti-patterns:
    - Don't ignore GoingAway (you'll be force-closed)
    - Don't reconnect without jitter (thundering herd)
    - Don't assume your last offset is still valid (retention may have advanced past it — check `SubscribeAck.start_offset` or `ReconnectAck.actual_start_offset`)
    - Don't send `Subscribe` after receiving `GoingAway` (you'll get `ServerDraining`)

- [x] **`docs/architecture.md`** — update with:
  - Server lifecycle state machine diagram (`Running → Draining → ShuttingDown`)
  - Drain sequence (force-flush → GoingAway → wait → force-close → engine shutdown)
  - Capability negotiation flow diagram

- [x] **`docs/design_decisions.md`** — document key decisions:
  - DD-020: GoingAway as separate frame type (not Error subcode) — Error frames trigger `is_fatal()` close; GoingAway needs a grace period
  - DD-021: Capability flags instead of version bump — allows gradual adoption and mixed client fleets
  - DD-022: Reconnect as optional optimization — `Connect + SubscribeAt` always works; Reconnect is for high-subscription-count / high-latency scenarios
  - DD-023: `drain_timeout` separate from `shutdown_timeout` — drain = client connections (variable), shutdown = storage flush (fixed)
  - DD-024: `/health` changed from always-200 to readiness-aware — draining server must not receive new traffic from LB
  - DD-025: Partial ReconnectAck — don't fail all subscriptions for one bad topic
  - DD-026: `GoingAway.deadline_ms` is a hint, not a contract — server may force-close early
  - DD-027: GoingAwayReason as newtype u8, not enum — forward compatibility; unknown reasons accepted, never rejected
  - DD-028: Capabilities before GoingAway in execution order — prevents 5,000 codec error logs per deploy with legacy clients
  - DD-029: Force-flush at drain entry — closes durability gap between last batch flush and SIGTERM
  - DD-030: Publishes always accepted during drain — rejecting them stalls the publish-subscribe pipeline mid-drain
  - DD-031: `admin_api_key` instead of `Permission::Admin` — admin operations are server-global, not namespace-scoped

---

## Dependency Graph

```
14.1 (drain state machine + health + capabilities)
  │
  ├──► 14.2 (GoingAway frame + ServerDraining error code)
  │       │
  │       ├──► 14.3 (Reconnect frame — optimization, ships after core validated)
  │       │
  │       └──► 14.4 (integration tests — core tests run without 14.3)
  │
  └──────────► 14.5 (operational tooling — admin drain API, metrics)
                │
                └──► 14.6 (documentation — captures final state)
```

**Recommended execution order:**

1. **14.1** first — everything depends on the lifecycle state machine, health endpoints, and capability negotiation. Capabilities MUST land before GoingAway so the server knows which clients can handle the new frame.
2. **14.2** next — GoingAway frame and ServerDraining error code. With 14.1 in place, the server sends GoingAway only to capable clients and falls back to `Error { InternalError }` for legacy clients. No codec errors, no regression.
3. **14.4** (core tests) — validate the deploy lifecycle end-to-end. Tests 14.4.1–14.4.6 and 14.4.10 run without Reconnect.
4. **14.3** after core is validated — Reconnect frame is an optimization. Ship it when the foundation is proven. Tests 14.4.7–14.4.8 activate.
5. **14.5** independent of 14.3 — operational tooling (admin drain API, metrics) can proceed in parallel with Reconnect work.
6. **14.6** last — documentation captures the final state of all phases.

## Config Summary

All new config fields with defaults:

```toml
[deploy]
drain_timeout_secs = 30                     # time for clients to disconnect voluntarily
shutdown_timeout_secs = 10                   # time for engine/storage flush after drain
going_away_reason = "upgrade"               # reason sent in GoingAway frame
# admin_api_key = ""                        # API key for /admin/* endpoints (>= 24 chars). If not set, admin endpoints return 403. Env: FAFB_ADMIN_API_KEY
```

## Protocol Summary

New wire types added by this plan:

| Type | Code | Direction | Phase | Purpose |
|---|---|---|---|---|
| `GoingAway` | 0x10 | Server → Client | 14.2 | Planned shutdown notification with deadline |
| `Reconnect` | 0x11 | Client → Server | 14.3 | Batch re-subscribe with offset resume |
| `ReconnectAck` | 0x12 | Server → Client | 14.3 | Per-subscription restore results |

Updated wire types:

| Type | Code | Change | Phase |
|---|---|---|---|
| `Connect` | 0x01 | Optional trailing `u32 capabilities` field | 14.1 |
| `ConnectAck` | 0x02 | Optional trailing `u32 server_capabilities` field | 14.1 |

New error code:

| Code | Value | Fatal | Phase | Purpose |
|---|---|---|---|---|
| `ServerDraining` | 0x000D | No | 14.2 | Request rejected because server is draining |

Capability bits (in `Connect` / `ConnectAck`):

| Bit | Name | Phase | Meaning |
|---|---|---|---|
| 0 | `CAP_GOING_AWAY` | 14.1 | Client/server understands GoingAway frame |
| 1 | `CAP_RECONNECT` | 14.1 | Client/server understands Reconnect/ReconnectAck frames |

## Dependencies Added

None. Capability flags use plain `u32` constants — no external crates required.

## Key Decisions to Record

| ID | Decision | Rationale |
|---|---|---|
| DD-020 | `GoingAway` is a distinct frame type, not an Error subcode | Error frames trigger `is_fatal()` close logic; GoingAway needs a grace period. Keeps error handling clean. |
| DD-021 | Capability flags instead of version bump | Version 2 would break all v1 clients simultaneously. Capabilities allow gradual adoption and mixed fleets. |
| DD-022 | `Reconnect` is optional (Connect + SubscribeAt always works) | Not all client SDKs need the optimization. Keeps the protocol simple for simple clients. Ships after core deploy story is validated. |
| DD-023 | `drain_timeout` separate from `shutdown_timeout` | Drain is about client connections (variable, depends on fleet size). Shutdown is about storage flush (fixed, depends on data volume). Different concerns, different knobs. |
| DD-024 | `/health` changed from always-200 to readiness-aware | A draining server must not receive new traffic from load balancers. Correct behavior > backward compatibility for health probes. |
| DD-025 | Partial `ReconnectAck` (per-subscription status) | Failing all 50 subscriptions because 1 topic was deleted during deploy is hostile. Let the client decide how to handle partial failures. |
| DD-026 | `GoingAway.deadline_ms` is a hint, not a contract | Server may need to shut down faster than advertised (e.g., cascading failure). Clients must handle early disconnection. |
| DD-027 | `GoingAwayReason` is a newtype `u8`, not a Rust enum | Forward compatibility — unknown reason values are accepted, never rejected. A future server can add reason `0x05` without breaking old clients. |
| DD-028 | Capabilities land before GoingAway in execution order | Without capabilities, sending GoingAway (0x10) to v1 clients causes 5,000 codec error logs per deploy. Capabilities let the server fall back to `Error { InternalError }` for legacy clients — no noise, no regression. |
| DD-029 | Force-flush storage at drain entry | Closes the durability gap between the last batch flush and SIGTERM. Up to 100ms / 4MB of writes could otherwise be lost if the process crashes during drain. One `engine.force_flush()` call, significant durability improvement. |
| DD-030 | Publishes always accepted during drain (no config flag) | Rejecting publishes stalls the publish-subscribe pipeline mid-drain. Every in-flight publish should complete so subscribers can ack before disconnecting. The added config flag would require its own test matrix for marginal value. |
| DD-031 | `admin_api_key` in `[deploy]` config instead of `Permission::Admin` | Admin operations are server-global, not namespace-scoped. Adding a new permission variant would touch auth, config, and every permission test. A single key comparison is simpler, more secure (separate from namespace keys), and doesn't pollute the permission model. |

## Done When

- [x] `ServerLifecycle` state machine drives all server behavior (accept, health, drain, shutdown) *(14.1)*
- [x] Capability negotiation in Connect/ConnectAck — backward compatible with v1 clients *(14.1)*
- [x] Server transitions through Running → Draining → ShuttingDown on SIGTERM *(14.1)*
- [x] Force-flush at drain entry guarantees no unflushed data loss *(14.1)*
- [x] Clients with `CAP_GOING_AWAY` receive `GoingAway` frame on drain *(14.2)*
- [x] Clients without `CAP_GOING_AWAY` receive `Error { ServerDraining }` on drain — no regression *(14.2)*
- [x] `/health/ready` returns 503 during drain, during init, and in shutting_down state *(14.1)*
- [x] `/health/live` returns 200 always *(14.1)*
- [x] HTTP server starts before engine.init() so health probes work during startup *(14.1)*
- [x] Publishes accepted during drain; subscribes rejected with `ServerDraining` *(14.2)*
- [x] Inflight timeout accelerated during drain *(14.1)*
- [x] Clients can `Reconnect` with batch subscriptions and offset resume in one round-trip (14.3)
- [x] `ReconnectAck` reports per-subscription status (restored, clamped, not_found, etc.) (14.3)
- [x] `POST /admin/drain` triggers drain via HTTP (14.5)
- [x] All integration test scenarios pass (14.4, 14.5)
- [x] Operator deploy guide covers single-instance and rolling deploy (14.6)
- [x] Client SDK guide covers GoingAway handling, Reconnect usage, offset tracking, backoff strategy (14.6)
- [x] Protocol docs updated with all new frame types, error codes, and capability bits (14.6)
- [x] All tests pass (`cargo test` — 485 tests, 0 failures), clippy clean *(14.5)*
