# Client SDK Deploy Guide

This guide is for SDK developers building client libraries that handle server deploys gracefully. It covers capability negotiation, GoingAway handling, offset tracking, reconnection strategies, and common pitfalls.

For the wire format specification, see [protocol.md](protocol.md). For operator deploy procedures, see [deploy.md](deploy.md).

## Capability Advertisement

Client SDKs should advertise supported capabilities in the Connect frame:

```
Connect {
    version: 1,
    namespace: "my-namespace",
    api_key: "my-api-key",
    capabilities: CAP_GOING_AWAY | CAP_RECONNECT   // 0x00000003
}
```

The server responds with the intersection of client and server capabilities:

```
ConnectAck {
    success: true,
    message: "",
    server_capabilities: 0x00000003   // server supports both
}
```

**Implementation notes:**

- Store the `server_capabilities` from ConnectAck for the session lifetime
- If `server_capabilities` includes `CAP_GOING_AWAY`, expect GoingAway frames during drain
- If `server_capabilities` includes `CAP_RECONNECT`, the Reconnect frame can be used after reconnecting
- If the server doesn't return capabilities (legacy server), treat `server_capabilities` as 0

**Capability constants:**

```
CAP_GOING_AWAY = 0x00000001   // bit 0
CAP_RECONNECT  = 0x00000002   // bit 1
```

## GoingAway Handling

When the server begins a graceful drain, capability-aware clients receive a `GoingAway` frame:

```
GoingAway {
    reason: 0x01,        // UPGRADE, MAINTENANCE, OVERLOADED, or SHUTDOWN
    deadline_ms: 28000,  // ~28 seconds until force-close
    message: "server upgrading to v2.1"
}
```

### Recommended SDK Response

Upon receiving GoingAway, the SDK should:

1. **Emit a `"going_away"` event** to the application with reason and deadline
2. **Stop sending new subscribe requests** — they will be rejected with `ServerDraining`
3. **Continue processing in-flight deliveries** — ack all messages currently being processed
4. **Unsubscribe from active topics** (optional but clean)
5. **Disconnect gracefully** via TCP FIN
6. **Schedule reconnection** with jitter (see [Reconnect Backoff](#reconnect-backoff))

```
// Pseudocode — recommended SDK implementation
on GoingAway(reason, deadline_ms, message):
    emit("going_away", { reason, deadline_ms, message })

    // Ack all in-flight messages
    for msg in inflight_messages:
        send(Ack { message_id: msg.id })

    // Unsubscribe from all topics
    for sub in active_subscriptions:
        send(Unsubscribe { topic: sub.topic, group: sub.group })

    // Disconnect
    close_connection()

    // Schedule reconnect with jitter
    delay = deadline_ms + random(0, min(5000, deadline_ms / 2))
    schedule_reconnect(delay)
```

### What Publishes Still Work

During drain, the server still accepts Publish frames and returns PublishAck. This allows in-flight publish-subscribe pipelines to complete. SDKs should NOT stop publishing just because GoingAway was received — let pending publishes complete naturally.

### Legacy Clients (No CAP_GOING_AWAY)

Clients that don't advertise `CAP_GOING_AWAY` receive `Error { code: ServerDraining, message: "server shutting down" }` instead of GoingAway. This error is **non-fatal** but the server will close the connection shortly. SDKs should treat it the same as an unexpected disconnect and use standard reconnection logic.

## Offset Tracking Requirement

**Clients MUST track the `offset` field from every `Deliver` frame they acknowledge.** This offset is required for resuming consumption after a disconnect.

```
// On every successful delivery:
on Deliver(message_id, offset, timestamp_millis, topic, payload):
    process(payload)
    send(Ack { message_id })

    // Critical: track the last acked offset per topic per group
    last_acked_offsets[topic][group] = offset
```

Without tracked offsets, clients cannot resume from where they left off after a deploy — they must either start from the head (missing everything published during the reconnect window) or from offset 0 (reprocessing everything).

### What to Store

Per active subscription, track:

| Field | Purpose |
|-------|---------|
| `topic` | Which topic this offset belongs to |
| `group` | Consumer group name (empty for individual) |
| `last_acked_offset` | The highest offset successfully acked |

### Persistence

For production use, persist offsets to survive client restarts:

- **In-memory** — simplest, but offsets lost on client restart (at-most-once)
- **Local file** — survives client restart (at-least-once with idempotent processing)
- **Database** — shared across client instances (recommended for consumer groups)

## Reconnect Strategies

After receiving GoingAway or experiencing a disconnect during a deploy, clients have two reconnection options.

### Strategy 1: Connect + SubscribeAt (Simple, Always Works)

Recommended for most clients, especially those with 1–5 subscriptions:

```
// 1. Reconnect with standard Connect
Connect { version: 1, namespace, api_key, capabilities }

// 2. Re-subscribe to each topic at last acked offset + 1
for (topic, group, offset) in saved_subscriptions:
    SubscribeAt { topic, offset: offset + 1, group }
    // Check SubscribeAck.start_offset — may differ from requested
```

Cost: `1 + 2N` round-trips for N subscriptions. At same-DC latency (<1ms RTT), this is imperceptible for 1–5 subscriptions.

### Strategy 2: Reconnect Frame (Optimized, High Subscription Count)

For clients with many subscriptions (50+) or high-latency connections:

```
// Single frame restores auth + all subscriptions
Reconnect {
    version: 1,
    namespace, api_key,
    capabilities: CAP_GOING_AWAY | CAP_RECONNECT,
    subscriptions: [
        { topic: "orders",  group: "",      last_acked_offset: 1042 },
        { topic: "events",  group: "etl",   last_acked_offset: 8876 },
        { topic: "metrics", group: "",      last_acked_offset: 55023 },
        // ... up to 10,000 subscriptions
    ]
}
```

Cost: 2 round-trips regardless of subscription count.

**Handling the response:**

```
on ReconnectAck(status, message, server_capabilities, results):
    if status == Success:
        // All subscriptions restored — delivery will begin
        for result in results:
            if result.status == OffsetClamped:
                log_warning("offset clamped for " + result.topic)
            update_offset(result.topic, result.group, result.actual_start_offset)

    elif status == Partial:
        // Atomic rollback — NO active subscriptions exist
        // Fall back to Connect + SubscribeAt for individual control
        log_warning("Reconnect partial failure: " + message)
        for result in results:
            log(result.topic + ": " + result.status)

        // Recover manually
        send(Connect { ... })
        for result in results:
            if result.status in [Restored, OffsetClamped]:
                send(SubscribeAt { topic: result.topic, offset: result.actual_start_offset, group: result.group })

    elif status == AuthFailed:
        // Auth failed — connection closed by server
        handle_auth_failure(message)
```

**Requirements:**
- Server must have advertised `CAP_RECONNECT` in a prior ConnectAck
- Maximum 10,000 subscriptions per Reconnect frame
- Must be sent in pre-auth state (before any Connect)

## Reconnect Backoff

### After GoingAway

Use the deadline from GoingAway plus jitter to avoid thundering herd:

```
reconnect_delay = deadline_ms + uniform_random(0, min(5000, deadline_ms / 2))
```

| drain_timeout | deadline_ms (approx) | jitter range | reconnect window |
|---------------|---------------------|--------------|------------------|
| 5s | ~5000 | 0–2500ms | 5.0–7.5s |
| 30s | ~30000 | 0–5000ms | 30–35s |
| 120s | ~120000 | 0–5000ms | 120–125s |

### On Connection Failure

After the initial reconnect attempt fails, use exponential backoff:

```
retry_delay = min(base_delay * 2^attempt, 30000)   // cap at 30 seconds
```

| Attempt | Delay (base=1000ms) |
|---------|-------------------|
| 1 | 1000ms |
| 2 | 2000ms |
| 3 | 4000ms |
| 4 | 8000ms |
| 5+ | 30000ms (capped) |

## Error Code Handling

### ServerDraining (0x000D)

- **Non-fatal** — connection remains open (briefly)
- Returned when sending Subscribe/SubscribeAt/SubscribeFrom/Reconnect during drain
- **Action:** Stop sending subscribe requests. Finish in-flight work. Disconnect.
- A well-behaved client that handles GoingAway immediately will never see this error

### Legacy Server (No Capabilities)

If the server doesn't return `server_capabilities` in ConnectAck (or returns 0):

- The server is pre-deploy (v1 without capability support)
- GoingAway frames will never be sent — shutdowns appear as unexpected disconnects
- Use standard error-recovery: reconnect with exponential backoff
- `Connect + SubscribeAt` is the only reconnection path (no Reconnect frame)

## Anti-Patterns

### Don't ignore GoingAway

If the SDK receives GoingAway and continues operating as normal, the server will force-close the connection after `deadline_ms`. In-flight messages may be lost. Always initiate graceful disconnect upon receiving GoingAway.

### Don't reconnect without jitter

If 5,000 clients all reconnect at exactly `deadline_ms`, the new server gets a connection spike that may exceed `max_connections`, causing `ResourceExhausted` errors for many clients. Always add randomized jitter.

### Don't assume your last offset is still valid

Retention may have advanced past the client's stored offset during the deploy window. Always check `SubscribeAck.start_offset` or `ReconnectAck.actual_start_offset` — they may be higher than expected. If `OffsetNotAvailable` is received, update the stored offset to the actual start position.

### Don't send Subscribe after receiving GoingAway

The server rejects new subscriptions during drain with `ServerDraining`. Don't waste round-trips — wait until reconnected to the new server.

### Don't use Reconnect on a legacy server

If the prior ConnectAck didn't include `CAP_RECONNECT` in `server_capabilities`, the server doesn't understand the Reconnect frame. Sending it will result in `Error { InvalidFrame }` and connection close. Always check capabilities before using Reconnect.

### Don't retry the same offset after OffsetClamped

If `ReconnectAck` reports `OffsetClamped` for a subscription, the original offset was deleted by retention. Don't retry with the old offset — use `actual_start_offset` from the response. The gap between your stored offset and the actual start represents messages that were deleted by retention.
