# Design Decisions Log

This document records cross-cutting design decisions that affect multiple subsystems. Each entry resolves an open question from one or more phase plans.

## DD-001: Auth is per-connection handshake, not per-message

**Decision:** Clients authenticate once via a Connect/ConnectAck handshake at connection start. All subsequent messages on that connection operate within the authenticated namespace.

**Rationale:** Per-message auth adds latency and complexity to every operation. The session model is standard for persistent TCP connections (MQTT, AMQP, Redis). A single namespace per connection simplifies the codec, session state, and permission checking.

**Affects:** Plan 3 (Codec), Plan 4 (Auth), Plan 8 (TCP Server)

## DD-002: No reply-to on Publish in v1

**Decision:** Publish messages do not carry a reply-to topic. Request-response patterns are not a protocol-level feature in v1.

**Rationale:** The namespace manager (Plan 7) has no concept of request-response routing. Adding reply-to adds codec complexity and implies the server must route responses, which is a significant scope increase. Clients can build request-response on top of topic naming conventions.

**Affects:** Plan 3 (Codec), Plan 7 (Namespace Manager)

## DD-003: Subscribe carries optional GroupName for consumer groups

**Decision:** The Subscribe and Unsubscribe frames include a `group` string field. Empty string means individual subscriber (broadcast). Non-empty means consumer group (competing consumers).

**Rationale:** Plan 7 defines two delivery modes (fan-out and consumer groups) that coexist per topic. The codec must carry the group field so the namespace manager can differentiate. An empty string is used rather than a separate flag byte to keep the frame format simple.

**Affects:** Plan 3 (Codec), Plan 7 (Namespace Manager), Plan 8 (TCP Server)

## DD-004: GroupName type must be added to types.rs

**Decision:** A `GroupName` newtype (validated non-empty when present, with `Display`/`FromStr`/`Serialize`/`Deserialize`) is added to `src/types.rs` as a prerequisite for codec implementation. This is a small amendment to the completed Phase 1.

**Rationale:** The codec Frame enum needs to represent consumer group names. Using raw `String` would bypass the validation pattern established by `NamespaceId`, `TopicName`, and `ApiKey`.

**Affects:** Plan 1 (Types — amendment), Plan 3 (Codec), Plan 7 (Namespace Manager)

## DD-005: PublishAck exists — publish is not fire-and-forget

**Decision:** The server responds to every Publish with a PublishAck containing the server-assigned `MessageId`. This confirms the message was accepted and persisted.

**Rationale:** Without PublishAck, clients have no backpressure signal and no confirmation of durability. Fire-and-forget contradicts the storage layer's durability guarantees. The assigned MessageId also lets clients correlate publishes with potential errors.

**Affects:** Plan 3 (Codec), Plan 7 (Namespace Manager), Plan 8 (TCP Server)

## DD-006: SubscribeAck and UnsubscribeAck exist

**Decision:** The server confirms Subscribe and Unsubscribe operations with explicit Ack frames.

**Rationale:** Without acks, a client sending Subscribe followed immediately by Publish has no way to know if its subscription was accepted. Race conditions between subscribe and message delivery become invisible. The ack frames echo back the topic and group for client-side correlation.

**Affects:** Plan 3 (Codec), Plan 7 (Namespace Manager), Plan 8 (TCP Server)

## DD-007: max_message_size is configurable via TcpConfig

**Decision:** `TcpConfig` has a `max_message_size` field (default 1,048,576 bytes / 1 MB) that the codec uses to reject oversized frames.

**Rationale:** Hardcoding the limit prevents operators from tuning it for their workload. The value needs to flow from config → codec, so it must be a config field.

**Affects:** Plan 2 (Config — amendment), Plan 3 (Codec)

## DD-008: Ping/Pong are empty frames

**Decision:** Ping and Pong carry no payload — they are just the message type byte with zero-length payload.

**Rationale:** Keepalive only needs to prove the connection is alive. Latency measurement or sequence numbers can be added in a future version if needed. Empty frames minimize overhead.

**Affects:** Plan 3 (Codec), Plan 8 (TCP Server)

## DD-009: Version negotiation is Connect-only, reject-on-mismatch

**Decision:** The protocol version byte is carried only in the Connect message. If the server does not support the requested version, it responds with an Error frame (code UnsupportedVersion) and closes the connection. There is no fallback/negotiation to a lower version.

**Rationale:** Version negotiation adds significant complexity (capability sets, feature flags). For a v1 protocol, a simple version check is sufficient. When v2 is introduced, the server can support both v1 and v2 concurrently and dispatch based on the Connect version byte.

**Affects:** Plan 3 (Codec), Plan 8 (TCP Server)

## DD-010: Deliver payload uses Bytes for zero-copy fan-out

**Decision:** The `Frame::Deliver` variant and internal payload representation use `bytes::Bytes` (reference-counted) rather than `Vec<u8>` for message payloads.

**Rationale:** When the server fans out a message to N subscribers, cloning a `Bytes` value is O(1) (atomic refcount increment) versus O(N * payload_size) for `Vec<u8>`. For a queue server handling large payloads with multiple subscribers, this is a critical performance optimization.

**Affects:** Plan 3 (Codec), Plan 7 (Namespace Manager), Plan 11 (Performance)

## DD-011: Ack uses MessageId, not TopicOffset

**Decision:** The `NamespaceManager::ack` method accepts a `MessageId` (not `TopicOffset`). The manager resolves the MessageId to a TopicOffset internally by scanning the client's inflight map.

**Rationale:** The wire protocol `Frame::Ack` carries a `MessageId`, not a `TopicOffset`. Requiring the TCP handler to maintain a separate `MessageId → TopicOffset` mapping pushes complexity into the wrong layer. The inflight maps are small (bounded by channel capacity), so the linear scan cost is negligible. A reverse subscription index provides O(1) lookup for which group/subscriber the client belongs to, avoiding the O(groups) scan.

**Affects:** Plan 7 (Namespace Manager), Plan 8 (TCP Server — simplifies ack handling)

## DD-012: Subscribe returns mpsc::Receiver

**Decision:** `NamespaceManager::subscribe` returns a `tokio::sync::mpsc::Receiver<PublishedMessage>`. The manager holds the `Sender` side and pushes messages via dedicated delivery tasks.

**Rationale:** This cleanly separates delivery scheduling (manager's concern) from message consumption (connection handler's concern). The TCP handler simply reads from the Receiver and encodes Deliver frames. The bounded channel provides natural backpressure — when a slow subscriber's channel fills, the delivery task retries with backoff rather than dropping messages or blocking other subscribers.

**Affects:** Plan 7 (Namespace Manager), Plan 8 (TCP Server — reads from Receiver in write loop)

## DD-013: Delivery tasks use Notify with pre-registration

**Decision:** Delivery tasks create a `Notify::notified()` future *before* calling `engine.consume_from()`, then only await it if consume returns `None`.

**Rationale:** Without pre-registration, a publish that calls `notify_waiters()` between the consume returning `None` and the task reaching `notified().await` would be lost, causing delayed delivery until the next publish. Pre-registration ensures the wake-up is captured even in this window. This is the standard pattern for `tokio::sync::Notify` in producer-consumer scenarios.

**Affects:** Plan 7 (Namespace Manager — delivery task correctness)

## DD-014: Queue engine config is server-level, not per-namespace

**Decision:** `max_depth`, `backpressure_policy`, `max_redelivery_attempts`, and `dlq_topic_suffix` are configured in a global `[queue]` section, not per-namespace.

**Rationale:** `DefaultNamespaceManager::new()` accepts a single value for `max_redelivery_attempts` and `dlq_topic_suffix` (not per-namespace). The original plan pulled these from the first namespace's config, but `HashMap::values()` iteration order is non-deterministic — with multiple namespaces, which config got used was effectively random. Promoting these to server-level config makes the behavior deterministic and explicit. Per-namespace overrides can be added as a future enhancement when the manager supports them.

**Affects:** Plan 2 (Config — `QueueConfig` added), Plan 6 (Queue Engine — `BackpressurePolicy` gains `Deserialize`), Plan 7 (Namespace Manager — constructor args), Plan 10 (Main — uses `config.queue.*`)

## DD-015: SIGTERM handling for container deployments

**Decision:** The main entrypoint handles both SIGINT (ctrl+c) and SIGTERM via `tokio::signal`. Both trigger the same graceful shutdown sequence.

**Rationale:** `tokio::signal::ctrl_c()` only catches SIGINT. Production deployments via Docker, Kubernetes, or systemd send SIGTERM for graceful shutdown (`docker stop`, `kubectl delete pod`, `systemctl stop`). Without SIGTERM handling, the process would be forcibly killed after the container runtime's grace period, bypassing storage flush and connection draining.

**Affects:** Plan 10 (Main)

## DD-016: Shutdown timeout prevents hung process

**Decision:** The graceful shutdown sequence (namespace manager + queue engine shutdown) is wrapped in a 10-second `tokio::time::timeout`. If the timeout expires, the process logs an error and force-exits with code 1.

**Rationale:** If a delivery task deadlocks or storage flush hangs, the process would never exit after receiving a shutdown signal. In container environments, this leads to the runtime's hard-kill timeout (typically 30s for Docker), which provides no diagnostic information. The explicit timeout provides a clear error log and deterministic exit behavior. 10 seconds is generous enough for normal shutdown (flush a few hundred segment files) but short enough to fail fast when something is genuinely stuck.

**Affects:** Plan 10 (Main)

## DD-017: Inflight entries have a timeout

**Decision:** Inflight entries older than 300 seconds (configurable via `inflight_timeout`) are evicted during ack processing before computing `minimum_live_offset`.

**Rationale:** Without a timeout, a subscriber that receives a message but never acks it would hold an inflight entry indefinitely, preventing compaction of that offset and all offsets below it. This would cause unbounded memory growth in the queue engine. The timeout acts as a safety valve — the message was already delivered to the subscriber's channel, so the inflight entry is purely for compaction tracking. For consumer groups, the DLQ mechanism handles redelivery concerns independently via `delivery_count`.

**Affects:** Plan 7 (Namespace Manager — compaction correctness), Plan 11 (Performance — memory management)

## DD-018: Constructor config structs replace parameter explosion

**Decision:** `InMemoryQueueEngine::new()` takes `(storage, EngineConfig)` and `DefaultNamespaceManager::new()` takes `(engine, ns_ids, quotas, ManagerConfig)`. Both config structs implement `Default`. The builder methods `.with_delivery_channel_capacity()`, `.with_max_consumer_groups_per_topic()`, and `.with_slow_consumer_config()` are removed.

**Rationale:** After plans 11.2–11.6, the engine constructor had 6 parameters and the manager constructor had 5 positional parameters plus 3 builder chains. This made call sites brittle (positional argument confusion) and every new feature required touching all call sites to add a parameter. Config structs with `Default` + struct update syntax (`..Default::default()`) make call sites self-documenting and new fields backward-compatible. `.with_interceptors()` is retained because interceptors are a different concern (trait objects, not scalar config).

**Affects:** All call sites in main.rs, tests, and benchmarks

## DD-019: Atomic CAS for per-namespace connection limits

**Decision:** Per-namespace connection limits use `AtomicUsize::fetch_update` (compare-and-swap loop) to atomically check the limit and increment the count in a single operation.

**Rationale:** The original implementation used a load-then-check-then-increment pattern, which has a TOCTOU race: two concurrent connections could both read the count as (limit - 1), both pass the check, and both increment — exceeding the limit by 1. The CAS loop ensures the check and increment are atomic: the closure returns `None` (rejecting the update) if the current value is at the limit, or `Some(current + 1)` otherwise. If another thread increments between the read and the CAS attempt, the loop retries with the new value.

**Affects:** Plan 11.3 (Connection Scaling — `NamespaceManager::connect()`)

## DD-020: QueueEngine::publish returns PublishResult with eviction count

**Decision:** `QueueEngine::publish()` returns `Result<PublishResult>` where `PublishResult { offset, evicted_messages }` instead of just `Result<TopicOffset>`. The namespace manager uses `evicted_messages` to adjust the per-namespace `message_count` during publish.

**Rationale:** Under DropOldest backpressure, `Topic::publish()` may evict one or more messages to make room. Previously, `message_count` was only decremented during ack-triggered compaction, not during publish-time evictions. This caused `message_count` to permanently drift upward: every eviction lost a decrement, eventually causing every namespace to permanently hit its `max_total_messages` quota even with active consumers. Threading the eviction count through the return type keeps the accounting accurate without adding a separate "eviction callback" mechanism.

**Affects:** Plan 6 (Queue Engine — `Topic::publish` return type), Plan 7 (Namespace Manager — quota tracking)

## DD-021: Timestamps are u64 milliseconds, not SystemTime

**Decision:** `MessageEnvelope.timestamp` is `u64` (milliseconds since Unix epoch), not `std::time::SystemTime`. A `now_millis()` helper function is the sole clock source.

**Rationale:** `SystemTime` has a non-portable bincode serialization (internal `Duration` layout varies across Rust/platform versions). `u64` millis is compact, deterministic, trivially comparable, and requires zero conversion at wire/HTTP/retention boundaries. Since the app has no deployed clients, this is a clean break with no migration cost.

**Affects:** Plan 12 (all phases — `MessageEnvelope`, wire protocol, HTTP API, storage format, retention)

## DD-022: Deliver frame includes offset and timestamp

**Decision:** `Frame::Deliver` now carries `offset: TopicOffset` (u64) and `timestamp_millis: u64` alongside the existing `message_id`, `topic`, and `payload`.

**Rationale:** Clients need offsets for tracking (resume after reconnect), debugging stale deliveries, and future consumer group rebalancing. Timestamps enable client-side latency measurement. Adding both now avoids a second breaking wire format change later. Cost is 16 bytes per delivery — negligible relative to typical payloads.

**Affects:** Plan 12 Phase 2 (Codec, namespace manager delivery path, TCP session)

## DD-023: Retention task scans storage directories, not in-memory engine

**Decision:** The retention task discovers topics via `storage.list_topics()` (filesystem scan), not `engine.list_topics()` (in-memory map).

**Rationale:** The engine's in-memory map only contains topics that are currently loaded. Topics compacted out of memory but with orphaned segments on disk would be missed by an engine-only scan. The filesystem is the authoritative source of what data exists. The scan runs at most once per `retention_check_interval_secs` (default 60s), so the I/O cost is negligible.

**Affects:** Plan 12 Phase 5 (`RetentionTask` implementation)

## DD-024: Retention floor is min_live_offset, not base_offset

**Decision:** Retention never deletes segments containing offsets that any active subscriber or consumer group still needs. `NamespaceManager::min_live_offset()` walks all subscriber and group cursors to find the minimum.

**Rationale:** `base_offset` reflects in-memory compaction, not subscriber needs. A subscriber at offset 800 with `base_offset` at 0 still needs offset 800. Deleting segments up to 1000 based on age would silently lose messages 800–999 for that subscriber. The `min_live_offset` floor prevents this. Slow consumers are handled separately via `SlowConsumerPolicy`.

**Affects:** Plan 12 Phase 5 (`NamespaceManager` trait, `RetentionTask` floor logic)

## DD-025: Compact-before-delete ordering in retention

**Decision:** The retention task always calls `engine.compact()` before `storage.delete_segments()` for a given topic within a single sweep iteration.

**Rationale:** If segments are deleted first and a crash occurs before compaction, the engine would attempt to replay deleted data on restart — data loss. If compaction happens first and a crash occurs before deletion, orphaned segment files remain on disk — harmless, cleaned up on the next sweep. The ordering is a hard invariant enforced in code, not a suggestion.

**Affects:** Plan 12 Phase 5 (`RetentionTask::sweep()` step ordering)

## DD-026: Per-namespace retention coexists with global segment cap

**Decision:** `StorageConfig.max_segments_per_topic` (global hard ceiling, enforced inline during rotation) and `NamespaceConfig.retention.max_segments` (per-namespace policy, enforced by background task) operate independently.

**Rationale:** The inline limit is a safety net that prevents runaway disk usage even if the retention task falls behind or is misconfigured. The retention task handles finer-grained per-namespace policy (age, bytes, count). Merging them would couple rotation-time behavior to background-task scheduling, creating a leaky abstraction.

**Affects:** Plan 12 Phase 4 (Config), Plan 12 Phase 5 (Retention task)

## DD-027: Dirty bytes tracked on Topic struct, not a separate map

**Decision:** The `batch_flush_size_bytes` feature tracks unflushed bytes as a `dirty_bytes: u64` field on `Topic`, not as a separate `Arc<RwLock<HashMap<..., AtomicU64>>>` map shared between publish and flush.

**Rationale:** The publish path already holds the topic mutex when calling `Topic::publish()`. Incrementing a field under that existing lock adds zero contention. The flush task also acquires the topic mutex briefly to call `take_dirty_bytes()` (read + reset). A separate concurrent map of atomics would add a second locking layer and a second HashMap keyed by the same `(NamespaceId, TopicName)` — duplicated state for no benefit. The simpler design has fewer allocations, fewer lock acquisitions, and is easier to reason about.

**Affects:** Plan 11.4 (Storage I/O — batch flush size trigger)

## DD-028: Parallel replay logs errors and continues

**Decision:** When `engine.init()` replays persisted topics in parallel, a failure in one topic (corrupt segment, I/O error) is logged at `error` level and skipped. The remaining topics continue replaying. A summary warning reports the count of failed topics.

**Rationale:** A single corrupt segment file should not prevent the entire server from starting. The affected topic will appear empty until re-published, which is safer than refusing to serve any traffic. Operators can inspect the error logs and recover the corrupt topic manually. The alternative (fail the entire init) would force manual data directory cleanup before the server can restart — a worse operational outcome.

**Affects:** Plan 11.4 (Storage I/O — parallel startup replay)

## DD-029: LRU fd management via lru::LruCache

**Decision:** `FsStorage` manages open segment file handles with `Mutex<LruCache<TopicKey, ...>>` bounded by `max_open_fds`, replacing the unbounded `RwLock<HashMap<...>>`.

**Rationale:** The previous design opened a segment file on first append and never closed it. With 10,000+ topics this would exhaust the process fd limit (typically 1024 on Linux). `LruCache::push()` automatically evicts the least-recently-used entry when at capacity, closing the file handle. `get()` updates access order so recently-written topics stay open. `peek()` is used in flush so background flushes don't artificially keep cold topics hot. A `Mutex` is used instead of `RwLock` because `LruCache::get()` requires `&mut self` to update access order — there's no read-only path.

**Affects:** Plan 11.4 (Storage I/O — file descriptor management)

## DD-030: Delivery task skip-forward on evicted offsets

**Decision:** When `engine.consume_from(offset)` returns `Ok(None)` and `offset < engine.base_offset()`, the delivery task skips forward to `base_offset` instead of waiting for a notify. A warning is logged and `fafb_messages_skipped_total` is emitted.

**Rationale:** Under DropOldest backpressure, in-memory eviction can move `base_offset` past a subscriber's cursor. The old code treated `Ok(None)` as "caught up" and waited for a notify, which woke the task on the next publish but the stale offset was still below `base_offset` — creating an infinite spin loop. The fix distinguishes "caught up" (`offset >= head_offset`) from "evicted" (`offset < base_offset`). This is a prerequisite for Plan 12 (Tiered Storage), where the cold-read path will be inserted between these two cases.

**Affects:** Plan 12 Phase A (Delivery task fix), Plan 7 (Namespace Manager — delivery correctness)

## DD-031: `disk_backed` as engine config, not `BackpressurePolicy` variant

**Decision:** `disk_backed` is a `bool` on `EngineConfig`. When true, the engine forces `DropOldest` for all topics and replaces the global memory budget gate with a disk capacity gate. `BackpressurePolicy` remains unchanged.

**Rationale:** A `BackpressurePolicy::Overflow` variant would behave identically to `DropOldest` inside `Topic::publish` — same eviction logic, same code paths. The only difference is at the engine level (skip memory check, check disk instead). A topic-level enum that controls engine-level behavior is a leaky abstraction. Keeping them separate means `BackpressurePolicy` semantics are self-contained, and `disk_backed` semantics are self-contained.

**Affects:** Plan 12 Phase D (Engine config, publish path)

## DD-032: `SegmentCursor` for sequential cold reads

**Decision:** `SegmentCursor` keeps the file handle and `BufReader` position open across sequential `read_next()` calls, giving O(1) per-message cost. The cursor auto-opens the next segment at boundaries.

**Rationale:** A naive `read_at` implementation does find-segment, open-file, scan-to-offset, close per message. A consumer catching up 10K messages behind would repeat this 10K times, each time reopening the same file and rescanning from the start. `SegmentCursor` turns O(n * records_in_segment) catchup into O(n). The delivery task already loops sequentially — the cursor matches its access pattern naturally.

**Affects:** Plan 12 Phase B (Storage cold-read path)

## DD-033: Segment index cache instead of per-read directory listing

**Decision:** Cache a sorted `Vec<(TopicOffset, PathBuf)>` per topic, populated on `init()`, updated on rotation and cleanup. `find_segment_for_offset` does a binary search — O(log segments) with no I/O.

**Rationale:** The alternative calls `fs::read_dir` on every cold read — a syscall per message for catching-up consumers. The cached index eliminates this. The index is small (one entry per segment) and updated only on rotation (append path) and cleanup (background task), both infrequent relative to reads.

**Affects:** Plan 12 Phase B (`FsStorage::read_at` implementation)

## DD-034: Cleanup task in `main.rs` queries `NamespaceManager` instead of engine-owned consumer offsets

**Decision:** Spawn a cleanup task in `main.rs` that queries `NamespaceManager::min_live_offset()` when it runs (every 60s). No new trait method on the engine, no hot-path overhead.

**Rationale:** The alternative adds `set_min_consumer_offset` to `QueueEngine`, called on every ack, maintaining a persistent map on the hot path for something that only matters once per minute. The namespace manager already tracks all subscriber and group cursors via `minimum_live_offset()`. Querying it periodically from the cleanup task avoids duplicating state and adding ack-path overhead.

**Affects:** Plan 12 Phase E (Segment deletion safety, `main.rs` wiring)

## DD-035: Integer percentage (`AtomicU32`) for disk usage cache instead of `f64`

**Decision:** Store disk usage as an integer percentage (0-100) in `AtomicU32`. The threshold comparison is `cached_pct < threshold_pct` — both integers, trivially correct.

**Rationale:** The alternative stores `f64` bits in `AtomicU64` via `to_bits()`/`from_bits()`. This works but is non-obvious — a reviewer has to reason about IEEE 754 bit representation to verify correctness. Integer percentage loses sub-percent precision, which is irrelevant for a capacity threshold (the difference between 89.7% and 90% does not matter for operational decisions).

**Affects:** Plan 12 Phase D (Disk capacity monitoring in engine)

## DD-036: Sparse offset index files (`.idx`) alongside segments

**Decision:** Write a 24-byte index entry every `index_interval` records (default 256) to a `.idx` file alongside each `.seg` file. The first record of every segment is always indexed. Binary search the in-memory index to seek within a segment instead of scanning from the start.

**Rationale:** `read_at` and `SegmentCursor::skip_to` were O(n) per seek where n is position within the segment. For a 64MB segment with ~640K records, seeking to the middle scanned ~320K records. The sparse index reduces this to O(log n + 256). Index files are small (~60KB per 64MB segment) and loaded entirely into memory. Missing indexes are rebuilt on startup.

**Affects:** Plan 13 Phase A (`FsStorage::read_at`, `SegmentCursor::skip_to`, `append()` index writes)

## DD-037: Index the first record of every segment

**Decision:** Initialize `records_since_last_index = index_interval` when creating a new segment so the very first record triggers an index entry at byte position 0.

**Rationale:** Without this, the first `index_interval` records in every segment had no index entry — `find_position()` returned `None` and fell back to scanning from byte 0. With first-record indexing, every seek benefits from the index regardless of target position within the segment.

**Affects:** Plan 13 Phase A (`ActiveSegment` initialization in `append()` and rotation)

## DD-038: Cache loaded `SegmentIndex` on `FsStorage` instead of loading from disk per call

**Decision:** Store loaded `SegmentIndex` instances in `RwLock<HashMap<PathBuf, Arc<SegmentIndex>>>` on `FsStorage`. `read_at` checks the cache before loading from disk. Cache entries are evicted when segments are deleted via `delete_segments`.

**Rationale:** Hot segments being read by multiple catching-up consumers would otherwise re-read the `.idx` file from disk on every `read_at` call. Index files are small (<100KB) so caching them has negligible memory impact. `SegmentCursor::skip_to` still loads from disk since cursors are short-lived and don't have access to `FsStorage`.

**Affects:** Plan 13 Phase A (`FsStorage::load_index`, `FsStorage::delete_segments`)

## DD-039: `RecordStream` tracks byte position internally

**Decision:** Added `byte_pos: u64` field to `RecordStream` that tracks cumulative bytes read. `position()` returns the current byte position; `seek_to_position()` updates it. This replaces external byte accounting (e.g., re-serializing envelopes to compute record size).

**Rationale:** `rebuild_index` was calling `bincode::serialize(&envelope)` on every record just to compute byte positions — wasteful and fragile (depends on bincode producing identical output). The record length is already encoded in the 4-byte prefix read by `next()`, so tracking cumulative position internally is both cheaper and more correct.

**Affects:** Plan 13 Phase A (`rebuild_index`, `RecordStream`)

## DD-040: Per-namespace disk quotas tracked at storage layer, enforced at manager layer

**Decision:** `FsStorage` tracks disk bytes per namespace incrementally using `RwLock<HashMap<String, AtomicU64>>`. The counter includes both `.seg` and `.idx` file sizes, updated on `init()`, `append()`, and `delete_segments()`. Quota enforcement happens in `DefaultNamespaceManager::publish()` — checking `NamespaceQuota.max_disk_bytes` against the tracked counter before calling `engine.publish()`.

**Rationale:** The storage layer is the source of truth for bytes-on-disk, so it owns the counter. The manager is the policy layer, so it owns enforcement — consistent with how `max_total_messages` and `max_topics` are enforced. Alternatives considered:

- *Engine-level enforcement* would require threading namespace config into the engine, violating its config-agnostic design.
- *Scanning disk on every publish* would add syscalls to the hot path; incremental tracking avoids this.
- *Exact enforcement via locking across the publish path* would prevent concurrent publishes; approximate TOCTOU enforcement (matching `max_total_messages` behavior) is acceptable since brief overshoot is harmless.

The `fafb_namespace_disk_bytes` gauge is emitted periodically by the cleanup background task (every 60s), not on every publish, to avoid adding an extra `await` to the hot path.

**Affects:** Plan 13 Phase B (`FsStorage`, `StorageBackend`, `QueueEngine`, `DefaultNamespaceManager`)

## DD-041: Segment compaction runs independently of retention with per-run error isolation

**Decision:** The compaction task (merge small segments) is a separate background task from the retention task (delete old segments). Each merge run is isolated — if one run fails, temp files are cleaned up, the error is logged, and remaining runs proceed.

**Rationale:** Compaction and retention solve different problems: compaction reduces file count and fd pressure, retention removes stale data. Coupling them would require shared discovery and careful ordering. Running independently means each can be tuned or disabled separately (`segment_merge_threshold_bytes = 0` disables compaction; `retention` section omitted disables retention).

Per-run error isolation ensures that a single corrupted or inaccessible segment doesn't prevent other topic segments from being merged. The compaction task always returns `Ok(merge_count)` to the background loop, never propagating individual run errors.

**Affects:** Plan 13 Phase D (`FsStorage::compact_segments`, compaction task in `main.rs`)

## DD-042: Layered tracing subscriber with optional file output

**Decision:** Use `tracing-subscriber`'s layered architecture with an `EnvFilter` wrapped in a `reload::Layer`, optional stdout and file output layers, and a non-blocking file writer via `tracing-appender`.

**Rationale:** The `tracing` ecosystem is already integrated throughout the codebase. A layered subscriber avoids replacing the framework. Non-blocking writes via `tracing_appender::non_blocking` push formatting to a background thread, keeping the publish/deliver hot path free of I/O. The `reload::Layer` enables dynamic log level changes at runtime via HTTP endpoint without restarting the server. `RollingFileAppender` provides time-based rotation (daily/hourly); a custom cleanup task enforces age and size limits since the appender doesn't support size-based rotation natively.

**Affects:** `main.rs` (`init_tracing`), `config.rs` (`LoggingConfig`), `server/http.rs` (admin endpoint)

## DD-043: Global panic hook for crash visibility

**Decision:** Set a global panic hook in `main()` that logs the panic message and source location through `tracing::error!` before delegating to the default hook.

**Rationale:** Tokio spawned tasks that panic produce a `JoinError` but the panic payload is often lost unless explicitly captured. By logging through tracing, panic information is captured in log files alongside normal operational logs, making post-mortem analysis possible. The hook preserves the default behavior (backtrace to stderr) by chaining to the previous hook.

**Affects:** `main.rs` (panic hook setup)

## DD-044: Background task respawn with exponential backoff

**Decision:** Monitor retention, cleanup, and compaction task `JoinHandle`s in a `loop { select! { ... } }`. Panicked tasks are respawned after an exponential backoff delay (1s, 2s, 4s, ... capped at 60s). Clean exits (e.g., compaction disabled) are logged and not respawned.

**Rationale:** A flat `select!` that falls through to shutdown on any background task exit leaves the server permanently degraded after a single transient panic. These tasks are stateless loops — safe to restart. Exponential backoff prevents spinning if the panic is deterministic. Capping at 60s bounds the maximum downtime between retries. TCP/HTTP server failures remain fatal since the server can't accept connections.

**Affects:** `main.rs` (monitoring loop, `backoff_delay` function)

## DD-045: Connection spans for log correlation

**Decision:** Each TCP connection is wrapped in `tracing::info_span!("conn", client, peer)` before spawning the connection handler task.

**Rationale:** With hundreds of concurrent connections, log lines from different clients interleave. Spans propagate through all tracing calls within the task, allowing operators to filter logs by client ID or peer address. This is zero-cost when the span is not sampled (tracing short-circuits disabled spans). The span is set at the TCP server level rather than inside the session to ensure it covers the full connection lifecycle including auth failures.

**Affects:** `server/tcp.rs` (accept loop, `Instrument` trait)

## DD-046: 80% threshold warnings with AtomicBool suppression

**Decision:** For resource limits where gradual growth is expected (message counts, disk bytes, connections), fire a `warn!` when usage crosses 80% of the configured limit. Suppress duplicate warnings via an `AtomicBool` flag that resets when usage drops back below the threshold.

**Rationale:** Hard limit warnings (at 100%) are reactive — operators learn about the problem at the same time clients start failing. An 80% early warning gives time to investigate and act. The suppression flag prevents log flooding under sustained pressure. The 80% threshold matches the existing pattern used for `global_max_memory_bytes`. `AtomicBool` with `Relaxed` ordering is sufficient since timing is not critical and the flag is only advisory.

**Affects:** `namespace/state.rs` (warning flags), `namespace/manager.rs` (publish/connect paths), `queue/engine.rs` (memory/disk checks)

## DD-047: Metrics for quota rejections, auth failures, and fd evictions

**Decision:** Add counters for events that were previously only logged or silently returned as errors: `fafb_quota_rejected_total`, `fafb_publish_rejected_total`, `fafb_auth_failures_total`, `fafb_fd_evictions_total`, `fafb_messages_evicted_total`.

**Rationale:** Logs are useful for debugging individual incidents, but metrics enable automated alerting and trend analysis. A Prometheus alert on `rate(fafb_quota_rejected_total[5m]) > 0` catches quota pressure immediately. Auth failure counters enable brute-force detection without parsing logs. Fd eviction counters show when the LRU cache is undersized. DropOldest eviction counters quantify data loss from backpressure.

**Affects:** `namespace/manager.rs`, `queue/engine.rs`, `queue/topic.rs`, `storage/fs.rs`, `auth/apikey.rs`

## DD-048: GoingAway is a distinct frame type, not an Error subcode

**Decision:** Introduce `GoingAway` (0x10) as a dedicated frame type rather than adding a new error code to the existing Error frame.

**Rationale:** Error frames have `is_fatal()` semantics — fatal errors trigger immediate connection close in the codec and session layers. GoingAway needs a grace period: the server sends it and then keeps the connection open for in-flight work to complete. Reusing the Error frame would require either making it non-fatal (breaking existing error handling) or adding "fatal-but-with-grace-period" semantics (adding complexity to every error handler). A separate frame type keeps error handling clean and makes GoingAway handling its own independent concern.

**Affects:** `codec/frame.rs`, `codec/encoder.rs`, `codec/decoder.rs`, `server/tcp/connection.rs`, `server/tcp/session.rs`

## DD-049: Capability flags instead of protocol version bump

**Decision:** Negotiate optional features via a `u32` capabilities bitmask in the Connect/ConnectAck handshake, rather than bumping the protocol version to 2.

**Rationale:** A version bump to 2 would break all v1 clients simultaneously — they would receive `Error { UnsupportedVersion }` and disconnect. Capabilities allow gradual adoption: a v1 client without capabilities connects normally (capabilities = 0), while a newer client can advertise support for GoingAway and Reconnect. Mixed client fleets work without coordination. The `u32` bitmask provides 32 feature flags with zero external dependencies. If the capability space exceeds ~8 bits in the future, a crate like `bitflags` can be considered.

**Affects:** `codec/frame.rs` (Connect/ConnectAck fields), `codec/encoder.rs`, `codec/decoder.rs`, `server/tcp/session.rs` (negotiation logic)

## DD-050: Reconnect as optional optimization, not required path

**Decision:** The Reconnect frame (0x11) is an optional protocol optimization. `Connect + SubscribeAt` always works and remains the primary reconnect path for all clients.

**Rationale:** Not every client SDK needs the optimization. A client with 1-5 subscriptions in a same-DC deployment (RTT < 1ms) has no perceptible benefit from Reconnect. Keeping it optional means simple SDKs can ignore Reconnect entirely and still participate in graceful deploys. The frame ships after the core deploy story (drain + GoingAway + health probes) is validated in production. Reconnect adds value for clients with many subscriptions (50+) or high-latency cross-region connections.

**Affects:** `codec/frame.rs`, `server/tcp/session.rs` (process_reconnect), `server/tcp/connection.rs` (drain guard)

## DD-051: drain_timeout separate from shutdown_timeout

**Decision:** Introduce `drain_timeout_secs` (default 30) as a separate config from `shutdown_timeout_secs` (default 10). Drain controls how long to wait for client connections. Shutdown controls how long for engine/storage flush.

**Rationale:** These are independent concerns with independent scaling characteristics. Drain timeout depends on client fleet size: 5,000 subscribers doing cold reads may need 30+ seconds, while a dev restart needs 0. Shutdown timeout depends on storage volume: fsyncing 100 active segments takes more time than fsyncing 2. A single timeout forces a compromise between "enough time for clients" and "enough time for storage." Separate knobs let operators tune each independently.

**Affects:** `config.rs` (DeployConfig), `main.rs` (shutdown sequence), `server/state.rs` (drain wait)

## DD-052: /health changed from always-200 to readiness-aware

**Decision:** Change `GET /health` from always returning 200 to returning 503 during drain, initialization, and shutdown. Add `GET /health/live` (always 200) for liveness probes and `GET /health/ready` for readiness probes. `/health` is an alias for `/health/ready`.

**Rationale:** A draining server must NOT receive new traffic from a load balancer. If `/health` always returns 200, the LB continues routing requests to a server that is about to shut down. Readiness-aware health probes are the standard mechanism for removing instances from rotation during deploys. Operators who need the old behavior can switch their probes to `/health/live`. The readiness probe also gates business routes during startup — `/health/ready` returns 503 until `engine.init()` completes.

**Affects:** `server/http/routes.rs` (health endpoints), `server/http.rs` (middleware)

## DD-053: Partial ReconnectAck with per-subscription status

**Decision:** When some subscriptions in a Reconnect frame fail (TopicNotFound, PermissionDenied, GroupFull), all successful subscriptions are atomically rolled back. The client receives `Partial` status with per-subscription details but zero active subscriptions.

**Rationale:** Keeping a partial set of subscriptions (e.g., 48 of 50) that the client didn't explicitly ask for is dangerous — the client doesn't know which ones are active and which failed. It may miss the failed topics entirely. Atomic rollback ensures the client always knows its state: either all subscriptions are active (Success) or none are (Partial). The client can then fall back to `Connect + SubscribeAt` for individual control over which subscriptions to restore. OffsetClamped is explicitly NOT a hard failure — it's an informational warning that doesn't trigger rollback.

**Affects:** `server/tcp/session.rs` (process_reconnect, rollback logic)

## DD-054: GoingAway.deadline_ms is a hint, not a contract

**Decision:** The `deadline_ms` field in GoingAway represents the remaining drain timeout at send time. The server may force-close the connection before the deadline expires.

**Rationale:** The server may need to shut down faster than advertised (e.g., cascading failure, operator sends SIGKILL). Clients must handle early disconnection gracefully regardless. Making `deadline_ms` a contract would require the server to guarantee it never closes before that time, constraining emergency shutdown paths. As a hint, it lets clients make smart reconnect timing decisions while preserving server autonomy. Client SDKs should use it for reconnect scheduling but never assume the connection will survive that long.

**Affects:** `server/tcp/connection.rs` (drain deadline), client SDK documentation

## DD-055: GoingAwayReason as newtype u8, not Rust enum

**Decision:** Define `GoingAwayReason` as a newtype wrapping `u8` with associated constants, rather than a Rust `enum` with `TryFrom`.

**Rationale:** A Rust enum with exhaustive matching would reject unknown reason values with a decode error. If a future server version sends reason `0x05`, older clients would fail to decode the GoingAway frame — exactly the scenario capabilities are designed to prevent. A newtype `u8` accepts any value and logs it. Client behavior is identical regardless of reason (ack inflight, disconnect, reconnect); the reason is purely informational for operator dashboards. This is the same forward-compatibility pattern used in many binary protocols (HTTP/2 error codes, QUIC frame types).

**Affects:** `codec/frame.rs` (GoingAwayReason type), `codec/decoder.rs`

## DD-056: Capabilities land before GoingAway in execution order

**Decision:** Capability negotiation (Phase 14.1) ships before GoingAway (Phase 14.2). The server cannot send GoingAway to any client until it knows whether that client can handle it.

**Rationale:** Without capabilities, sending GoingAway (message type 0x10) to a v1 client causes a codec error — the client doesn't recognize the frame type and disconnects with an error log. With 5,000 connected clients, that's 5,000 spurious error logs on every deploy, masking real failures and generating false alerts. Capabilities let the server check `CAP_GOING_AWAY` before deciding what to send: GoingAway for capable clients, `Error { ServerDraining }` for legacy clients. No noise, no regression.

**Affects:** Implementation ordering (14.1 before 14.2)

## DD-057: Force-flush storage at drain entry

**Decision:** Call `engine.force_flush()` immediately upon entering drain mode, before sending GoingAway to any client.

**Rationale:** The batch flush runs every 100ms by default, but during the window between the last flush and the SIGTERM, up to 100ms / 4MB of writes may exist only in memory. If a client reconnects to a new instance that reads from the same disk, those unflushed messages must be there. One explicit `force_flush()` at drain entry closes this durability gap. The cost is one synchronous flush call — negligible compared to the drain timeout that follows. Without this, a fast drain (timeout = 0) could lose the most recent writes.

**Affects:** `main.rs` (shutdown sequence), `queue/engine.rs` (force_flush), `storage/fs.rs` (flush_all)

## DD-058: Publishes always accepted during drain

**Decision:** The server accepts Publish frames during drain mode. There is no config flag to reject publishes during drain.

**Rationale:** Rejecting publishes stalls the publish-subscribe pipeline mid-drain. A subscriber with in-flight messages needs the publisher to complete its work so the subscriber can ack and disconnect. If publishes are rejected, the subscriber's in-flight messages time out, get redelivered (or DLQ'd), and the drain takes longer than necessary. Every in-flight publish should complete so the subscriber can ack before disconnecting. An added `accept_publishes_during_drain` config flag would require its own test matrix and error handling in client SDKs for marginal value.

**Affects:** `server/tcp/session.rs` (Draining state frame dispatch)

## DD-059: admin_api_key in [deploy] config instead of Permission::Admin

**Decision:** Admin endpoints (`/admin/*`) use a dedicated `admin_api_key` in the `[deploy]` config section rather than adding a `Permission::Admin` variant to the existing permission system.

**Rationale:** Admin operations (triggering drain, viewing server state) are server-global — they don't belong to any namespace. Adding `Permission::Admin` would touch the Permission enum, auth module, config parsing, and every permission test. The existing permission model is namespace-scoped by design: an API key grants permissions within a single namespace. Admin operations cross all namespaces. A single key comparison with constant-time equality (`subtle::ConstantTimeEq`) is simpler, more secure (admin key is separate from namespace keys), and doesn't pollute the permission model. The key supports env override (`FAFB_ADMIN_API_KEY`) for container deployments.

**Affects:** `config.rs` (DeployConfig), `server/http/routes.rs` (admin auth middleware), `server/http.rs` (AppState)
