# Delivery Guarantees

A definitive reference for fafb-q's delivery semantics. Read this before building production consumers — understanding when duplicates happen and what "durable" means in each mode will save you from surprises.

## At a Glance

| Property | fafb-q | Kafka (`acks=1`) | NATS JetStream | RabbitMQ (confirms + ack) |
|----------|--------|-------------------|----------------|---------------------------|
| Delivery guarantee | At-least-once | At-least-once | At-least-once | At-least-once |
| Exactly-once | No (application-level) | Yes (with idempotent producer + transactions) | No (application-level) | No (application-level) |
| PublishAck means | Page cache write | Page cache write (leader only) | Replicated to R nodes | Written to disk (with confirms) |
| Survives process crash | Yes | Yes | Yes | Yes |
| Survives OS crash | Only with `every_write` fsync | Only with `acks=all` + `min.isr` | Yes (R > 1) | Yes (with `fsync`) |
| Consumer offset storage | Client-managed | Server-managed (`__consumer_offsets`) | Server-managed | Server-managed |
| Redelivery trigger | Inflight timeout (300s default) + exponential backoff; dead-client detection via idle timeout (30s default) | Consumer poll timeout | Ack wait timeout | Consumer nack / timeout |
| Dead letter queue | Built-in (after max attempts) | No (application-level) | Yes (max deliveries) | Yes (reject + DLQ exchange) |
| Ordering | Per-topic, per-connection | Per-partition | Per-stream | Per-queue |

**The headline:** fafb-q provides **at-least-once delivery**. Every published message is delivered to each subscriber at least once, provided the subscriber connects before retention removes it. Duplicates are possible — the three scenarios are documented below. Exactly-once delivery requires application-level idempotency.

---

## Publisher Guarantees

### What PublishAck means

When your client receives a `PublishAck`, the message has been:

1. **Assigned a unique, sequential offset** within `(namespace, topic)`
2. **Added to the in-memory buffer** (available for immediate delivery to push subscribers)
3. **Appended to the segment file** via the OS page cache (`write_all()`)

With the default fsync mode (`every_batch`), the message has **NOT** been fsynced to disk at this point — fsync happens in the background. With `every_write` mode, `sync_all()` is called inside `storage.append()` before returning, so PublishAck **does** mean "fsynced to disk."

> **Why this matters:** A fafb-q **process crash** (segfault, OOM kill, `kill -9`) does NOT lose your message — the OS page cache survives process death, and the kernel will eventually flush it to disk. An **OS crash or power loss** before fsync completes may lose recent messages (unless you use `every_write` mode). This is the same guarantee as Kafka with `acks=1` (single broker, no replication).

### Durability by fsync mode

| Mode | Config value | When fsync happens | Process crash | OS crash / power loss |
|------|-------------|-------------------|---------------|----------------------|
| **Every batch** (default) | `fsync = "every_batch"` | Background flush task, every `batch_flush_interval_ms` (default 100ms) or when dirty bytes exceed `batch_flush_size_bytes` (default 4MB) | Safe | May lose up to ~100ms of writes |
| **Every write** | `fsync = "every_write"` | `sync_all()` called inside `storage.append()`, before PublishAck is sent | Safe | **Safe** — PublishAck = fsynced to disk |
| **OS managed** | `fsync = "os_managed"` | Never explicitly called; relies on kernel writeback (typically 5-30s) | Safe | May lose seconds of writes |

Choose your mode based on your durability/throughput trade-off:
- **Most applications:** `every_batch` (default). Page cache survives process crashes. The ~100ms fsync window is acceptable for nearly all workloads.
- **Financial / audit data:** `every_write`. Every message is fsynced before PublishAck. Lower throughput but maximum durability.
- **Metrics / telemetry:** `os_managed`. Highest throughput. Acceptable to lose a few seconds of data on OS crash.

### Offset assignment

- Offsets are **monotonically increasing, gap-free, and unique** within a `(namespace, topic)`.
- Once assigned, an offset is **never reused** — not by compaction, not by retention, not by restart.
- After a server restart, offset assignment continues from the last persisted message's offset + 1.
- Publishers can use offsets for deduplication and progress tracking.

### Ordering

- Messages published on a **single TCP connection** to a **single topic** are stored in send order. The offset order matches the send order.
- Messages from **different connections** or to **different topics** have no ordering guarantee relative to each other.
- HTTP publishes are also ordered per-connection (one at a time), but concurrent HTTP requests from different clients have no mutual ordering.

---

## Push Subscriber Guarantees (Subscribe / Deliver / Ack)

### At-least-once delivery

Every message published to a subscribed topic will be delivered to the subscriber **at least once**, provided:

1. The subscriber maintains or re-establishes a connection before retention removes the message.
2. The subscriber's channel has capacity (the server can reach the client).

If a message cannot be delivered (channel full, client disconnected), the server retries. After `max_redelivery_attempts` (default 5) failed deliveries, the message is routed to the dead letter queue.

### When duplicates happen

There are exactly three scenarios where a subscriber receives the same message more than once. Understanding these lets you build idempotency only where it matters.

#### 1. Inflight timeout

```
Client                          Server
  |                               |
  |  <-- Deliver (offset=5) ---   |  Server starts inflight timer
  |                               |
  |  ... client processing ...    |
  |                               |  Timer expires (inflight_timeout_secs, 300s default)
  |                               |  Server redelivers offset=5
  |  <-- Deliver (offset=5) ---   |
  |                               |
  |  --- Ack (offset=5, 1st) -->  |  Ack accepted, removed from inflight
  |  --- Ack (offset=5, 2nd) -->  |  Ack ignored (not in inflight)
```

The client processes the message twice. The second ack is a harmless no-op.

**Redelivery backoff:** Each redelivery uses exponential backoff. The first attempt waits `inflight_timeout_secs` (default 300s). Subsequent attempts double: 300s → 600s → 1200s → 2400s → capped at `max_redelivery_backoff_secs` (default 3600s). A ±25% random jitter prevents thundering herd when many messages time out simultaneously. After `max_redelivery_attempts` (default 5), the message is routed to the dead-letter queue.

With default settings, the total time before DLQ routing is approximately: 300 + 600 + 1200 + 2400 + 3600 ≈ 2.3 hours (varies with jitter).

**Configuration:**

```toml
[queue]
inflight_timeout_secs = 300        # Base timeout before first redelivery
max_redelivery_backoff_secs = 3600 # Maximum backoff cap (1 hour)
max_redelivery_attempts = 5        # Attempts before DLQ routing
```

**Mitigation:** Ack messages promptly. If processing takes longer than the inflight timeout, increase `inflight_timeout_secs`.

#### 2. Reconnect overlap

```
Client                          Server
  |                               |
  |  <-- Deliver (offset=5) ---   |
  |                               |
  |  ... connection drops ...     |
  |                               |  Server detects disconnect,
  |                               |  clears inflight for this client
  |                               |
  |  --- Reconnect ------------> |  SubscribeAt(last_acked=4)
  |                               |  Resumes from offset 5
  |  <-- Deliver (offset=5) ---   |  Duplicate: offset 5 delivered again
```

The client received offset 5 but never acked it. On reconnect, it resumes from `last_acked_offset + 1 = 5`, so offset 5 is delivered again.

**Mitigation:** This is by design — at-least-once requires redelivery of unacked messages after disconnect. Make your processing idempotent for this window.

#### 3. Consumer group rebalance

```
Consumer A                      Server                      Consumer B
  |                               |                            |
  |  <-- Deliver (offset=5) ---   |                            |
  |                               |                            |
  |  ... A disconnects ...        |                            |
  |                               |  A's inflight cleared      |
  |                               |  offset=5 reassigned       |
  |                               |                            |
  |                               |  --- Deliver (offset=5) -> |
  |                               |                            |
```

Consumer A received offset 5 but disconnected before acking. The server removes A's inflight entries and redelivers to Consumer B. If A already processed the message, it's a duplicate from the application's perspective.

**Mitigation:** Use message_id-based deduplication in your consumer logic when processing has side effects.

### Ordering

- Messages are delivered in **offset order** within a topic on a given connection.
- **Redelivered messages arrive out of order.** If offset 5 times out and is redelivered while offset 10 has already been delivered, offset 5 arrives after offset 10.
- Applications that require **strict ordering** must: (1) process one message at a time, (2) ack before requesting the next, (3) accept that redelivery breaks the ordering guarantee.

### Fan-out independence

Each non-group subscriber has its own delivery state (next offset, inflight map, delivery channel). One subscriber being slow, disconnected, or failing does not affect delivery to other subscribers on the same topic.

### Delivery channel backpressure

Each subscriber has a bounded delivery channel (`delivery_channel_capacity`, configured per connection). When the channel is full, the server pauses delivery for that subscriber until space opens. Other subscribers are unaffected.

---

## Consumer Group Semantics

Consumer groups provide **competing consumer** semantics: each message is delivered to exactly one group member.

### Delivery model

- Messages are distributed **round-robin** across active group members.
- Each message is tracked in the group's **inflight map** until acked.
- The group's `next_offset` advances on *delivery* (so the next message goes to the next consumer), while the **inflight map** ensures unacked messages are redelivered. The effective "committed offset" is the lowest offset still in the inflight map — this is what `min_live_offset()` computes to gate retention and segment cleanup.

### Redelivery within groups

- If a group member does not ack within the inflight timeout, the message becomes eligible for redelivery to **any** active group member (not necessarily the original).
- After `max_redelivery_attempts` (default 5) failed deliveries, the message is routed to the **dead letter queue** (DLQ topic: `{topic}{dlq_suffix}`).
- DLQ routing preserves the original payload and adds `_fafb.dlq.*` system headers with provenance metadata (source topic, original offset, delivery count, reason, timestamps).

### State on restart

**Consumer group state is entirely in-memory.** On server restart:

- All group memberships are lost.
- All inflight entries are lost.
- All committed offsets are lost.

Clients must reconnect and use `SubscribeAt` (or `Reconnect` frame) with their last-acked offset + 1 to resume. The server does not persist consumer group offsets.

> **Comparison with Kafka:** Kafka stores consumer group offsets in the `__consumer_offsets` topic, so consumers can resume after broker restart without tracking offsets themselves. fafb-q does not have this — your client or application must store its last-acked offset externally if you need crash-resilient resumption.

### Disconnect behavior

When a group member disconnects:

1. The member is removed from the group's member list.
2. All inflight entries assigned to that member are **removed from the inflight map** — these offsets become eligible for redelivery to remaining members.
3. If the group is now empty, the group enters an `emptied` state but is not immediately deleted.

---

## Fetch Semantics (Pull)

Fetch is a **stateless, pull-based** read mechanism. It is fundamentally different from Subscribe/Deliver.

### No delivery guarantee

The server maintains **no state** for fetch consumers. Each fetch request is independent:
- No inflight tracking.
- No redelivery on failure.
- No consumer group coordination.
- No offset persistence.

If your client crashes between fetching and processing, you must re-fetch. This is by design — fetch consumers manage their own reliability.

### Consistency

- `Fetch(topic, offset, max_count)` returns the message at `offset` if it exists and has not been removed by retention.
- The response includes `head_offset` (latest available) and `next_offset` (for pagination). `has_more = next_offset < head_offset`.
- If the requested offset is below the oldest retained offset, the server clamps to the oldest available and includes an `OffsetNotAvailable` warning.

### When to use Fetch vs. Subscribe

| Use case | Recommendation |
|----------|---------------|
| Real-time processing, low latency | Subscribe (push) |
| Batch jobs, periodic processing | Fetch (pull) |
| Messages > 100KB | Fetch (pull) — larger response budget via HTTP |
| Serverless / Lambda | Fetch via HTTP (pull) |
| Need server-side redelivery | Subscribe (push) |
| Simple offset management | Fetch (pull) |

See [fetch_consumer_guide.md](fetch_consumer_guide.md) for detailed patterns.

---

## Building Idempotent Consumers

Since fafb-q provides at-least-once delivery, your consumers must handle duplicates gracefully. Here are three patterns, from simplest to most robust.

### Pattern 1: Dedup by message_id

Every message has a unique `message_id` assigned by the server. Track processed IDs in a set (in-memory or external store) and skip duplicates.

```
on_message(msg):
    if msg.message_id in processed_set:
        ack(msg)  # Already processed, just ack
        return
    process(msg)
    processed_set.add(msg.message_id)
    ack(msg)
```

**Trade-off:** The processed set grows unboundedly. Use a time-windowed set (e.g., keep IDs for 2x the inflight timeout) or a probabilistic structure (bloom filter) for high-volume topics.

### Pattern 2: Idempotent writes

Design your processing so that applying the same message twice produces the same result. Examples:
- **Database upsert:** `INSERT ... ON CONFLICT DO UPDATE` with the message offset as a unique key.
- **File write:** Write to a path derived from the message offset — writing the same content twice is a no-op.
- **Counter increment:** Instead of `counter += 1`, store `counter = max(counter, msg.value)`.

This is the most robust pattern — no external state needed for dedup.

### Pattern 3: Exactly-once via external store

Combine message processing and offset tracking in a single transaction:

```
on_message(msg):
    begin_transaction()
    if get_last_offset(topic, group) >= msg.offset:
        rollback()  # Already processed
        ack(msg)
        return
    process(msg)  # Business logic within transaction
    set_last_offset(topic, group, msg.offset)
    commit_transaction()
    ack(msg)
```

This gives you **effectively exactly-once processing** as long as your business logic and offset storage share the same transactional store (e.g., same database).

---

## Failure Mode Matrix

What happens when things go wrong — verified against the actual code behavior.

### Server-side failures

| Failure | Impact | Recovery | Data loss? |
|---------|--------|----------|------------|
| **fafb-q process crash** (kill -9, OOM kill, panic) | All connections drop. In-memory state lost (consumer groups, inflight, subscriptions). | Restart. Topics replayed from disk. Clients reconnect with `SubscribeAt` at last-acked + 1. | No — page cache survives process death. Kernel flushes to disk. |
| **OS crash / power loss** | Same as process crash, plus page cache lost. | Restart OS and fafb-q. Topics replayed from disk. | **Possible** — messages written to page cache but not yet fsynced are lost. Window: ~100ms with `every_batch`, ~0 with `every_write`, up to 30s with `os_managed`. |
| **Disk full** | New publishes rejected with `ResourceExhausted` error. Existing data intact. Warning fired at 80% usage. | Free disk space or increase `disk_capacity_threshold_pct`. Publishes resume automatically. | No — rejection prevents partial writes. |
| **Out of memory** | In normal mode: publishes rejected when `global_max_memory_bytes` exceeded. In disk-backed mode: oldest messages evicted from in-memory buffer (`DropOldest`). | Reduce message rate, increase memory budget, or enable disk-backed mode. | No data loss on disk. In-memory eviction means those messages must be read from cold storage (disk) — slower but available. |
| **Single topic panic** | The panicking connection's task crashes. Other connections unaffected (crash isolation). Global panic hook logs the panic. | Connection auto-closes. Client reconnects. Background tasks respawn with exponential backoff. | No — panic in one connection does not affect storage or other connections. |

### Client-side failures

| Failure | Impact | Recovery | Duplicates? |
|---------|--------|----------|-------------|
| **Client crash mid-processing** (after Deliver, before Ack) | Message stays in server's inflight map until timeout expires. | Client reconnects, subscribes at last-acked + 1. Server redelivers the unacked message. | **Yes** — if the client partially processed the message before crashing. |
| **Client disconnect (clean)** | Unsubscribe sent. Server clears client's subscriptions and inflight entries. | Client reconnects when ready. | No — clean disconnect clears state. |
| **Client disconnect (unclean — network drop)** | Server detects via idle timeout (30s default — client keepalive Pings stop arriving). Clears subscriptions and inflight. For consumer groups, inflight messages are released for redelivery to other members. | Client reconnects. For groups: inflight messages reassigned to other members. | **Possible** — if another group member receives a message the disconnected client already processed. |
| **Slow consumer** | Delivery channel fills up. Server pauses delivery for this consumer. Other consumers unaffected. Retention may be clamped if this consumer holds the minimum live offset. | Speed up consumer or increase `delivery_channel_capacity`. | No — messages are delayed, not lost. Retention warns when a slow subscriber prevents segment cleanup. |

### Network failures

| Failure | Impact | Recovery | Duplicates? |
|---------|--------|----------|-------------|
| **Network partition (client can't reach server)** | Client sees connection timeout/reset. Server sees idle timeout (~30s). | Both sides clean up independently. Client reconnects when partition heals. | **Possible** — same as unclean disconnect. Messages delivered before partition may be redelivered after reconnect. |
| **Packet loss / reordering** | TCP handles retransmission. Application-level framing ensures complete messages. | Transparent — TCP guarantees in-order, reliable delivery at transport layer. | No — TCP handles this. |

### Storage-layer failures

| Failure | Impact | Recovery | Data loss? |
|---------|--------|----------|------------|
| **Corrupt segment file** | Topic replay may fail on restart for that topic. Error logged and topic skipped (partial startup). | Investigate and repair or delete corrupt segment. Other topics unaffected. | **Possible** — messages in the corrupt segment may be lost. |
| **Missing index file** | Index is rebuilt from segment data on startup (`rebuild_index()`). Slower startup but no data loss. | Automatic — index files are derived data. | No. |
| **Segment write failure** (permission denied, I/O error) | The in-memory publish happens *before* `storage.append()`. If the storage write fails, consumers may read a message that won't survive restart. The publish returns an error to the client (no PublishAck sent), but the message is already in the in-memory buffer. | Fix the underlying I/O issue. The message exists in-memory until evicted or server restarts. | **Possible** — message visible to in-memory consumers but not on disk. Lost on restart. This is a deliberate trade-off: the reverse ordering (persist first) would risk offset gaps on storage failure. Page cache writes virtually never fail in practice. |

---

## Delivery Observability Metrics

fafb-q exposes Prometheus metrics for monitoring delivery health:

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `fafb_consumer_lag` | Gauge | `namespace`, `topic`, `group` | Unprocessed messages per consumer group: `head_offset - next_offset + inflight_count`. Updated on every publish (head advances) and every ack (inflight clears). |
| `fafb_ack_latency_seconds` | Histogram | `namespace`, `topic` | Time between message delivery and ack. Shows consumer processing time distribution. Recorded on each successful ack (single and batch). |
| `fafb_duplicate_ack_total` | Counter | `namespace`, `topic` | Acks received for message IDs not found in inflight (already acked or expired). Useful for detecting client-side double-ack bugs or aggressive retry. |
| `fafb_redelivery_total` | Counter | `namespace`, `topic` | Messages redelivered after inflight timeout (exponential backoff). |
| `fafb_redelivery_delay_seconds` | Histogram | — | Actual backoff delay before each redelivery. Shows backoff distribution. |
| `fafb_batch_ack_size` | Histogram | — | Number of message IDs per BatchAck frame. |
| `fafb_delivery_attempts` | Histogram | `namespace`, `topic`, `outcome` | Delivery count at time of resolution (outcome: ack/nack/dlq). |

**Cardinality note:** `fafb_consumer_lag` labels are `{namespace, topic, group}`. In deployments with many namespaces, topics, and consumer groups, this may produce a high number of time series. Monitor cardinality and consider filtering or aggregation in your metrics pipeline if needed.

## Summary of Guarantees

| Guarantee | Status |
|-----------|--------|
| At-least-once delivery (push subscribers) | Yes |
| At-least-once delivery (fetch consumers) | No — client-managed |
| Exactly-once delivery | No — requires application-level idempotency |
| Offset continuity (no gaps, no reuse) | Yes |
| Ordering within topic + connection | Yes |
| Ordering across connections | No |
| Ordering preserved during redelivery | No |
| Publish durability (survives process crash) | Yes |
| Publish durability (survives OS crash) | Depends on fsync mode |
| Consumer group offset persistence across restart | No — client must track |
| Fan-out subscriber independence | Yes |
| Dead letter queue after max attempts | Yes |
| Per-message TTL (time-to-live) | Yes — expired messages skipped at delivery/fetch |
