# CLAUDE.md

## Project Overview

**fafb-q** is a centralized queue server written in Rust. Clients connect to subscribe and publish on queues. The server exposes:

- **TCP interface** — for subscribe and publish, using a custom messaging protocol defined by this project
- **HTTP interface** — for publish and queue information queries

The system is organized around **namespaces**. Each namespace:
- Has an API key with permissions (subscribe, publish, query, fetch)
- Contains any number of **queue topics**
- Is managed by a **namespace manager** that tracks consumers and subscribers

Queue topics are buffered in-memory and synced to filesystem for durability.

## Documentation and Plans

- `./docs/` — all documentation
- `./plans/` — all implementation plans

## Build Commands

```bash
cargo build                    # debug build
cargo build --release          # release build
cargo test                     # run all tests
cargo test <test_name>         # run a single test by name
cargo test -- --nocapture      # run tests with stdout visible
cargo clippy                   # lint
cargo fmt                      # format code
cargo fmt -- --check           # check formatting without modifying
```

Rust edition: **2024**

## Doc Maintenance

After every change, feature, or refactor — evaluate whether `README.md`, `docs/`, or `config/config.example.toml` need updating.

## Architecture Principles

- **Modularity**: Each subsystem is its own module with clean trait boundaries. Depend on traits, not concrete types.
- **Testability**: Every module testable in isolation. Use trait objects or generics for dependency injection. Integration tests in `tests/`, unit tests inline with `#[cfg(test)]`.
- **Protocol documentation**: The TCP messaging protocol is a first-class deliverable. Protocol changes must update docs alongside code.

## Module Map

| Path | Responsibility |
|---|---|
| `src/main.rs` | Binary entry point, server startup (`run_server`), CLI dispatch, background task supervision |
| `src/config.rs` | `ServerConfig`, `ServeArgs` (CLI flags), all TOML config structs, `RuntimeConfig` for hot-reload |
| `src/codec/` | TCP protocol codec — `MessageCodec`, `Frame` enum (all frame types), encoder/decoder |
| `src/server/tcp/` | TCP accept loop, `run_connection()`, `Session` state machine (PreAuth→Authenticated→Draining) |
| `src/server/http/` | Axum HTTP server — business routes (`/v1/*`), admin routes (`/admin/*`), health endpoints |
| `src/server/state.rs` | `ServerLifecycle` — atomic state machine (Running→Draining→ShuttingDown) |
| `src/namespace/` | `DefaultNamespaceManager` — subscriptions, consumer groups, delivery tasks, DLQ routing, inflight tracking |
| `src/queue/` | `InMemoryQueueEngine` — per-topic buffer, publish, consume, compaction, memory accounting |
| `src/storage/` | `FsStorage` — segment files (`.seg`+`.idx`), LRU fd cache, replay, cold reads, compaction |
| `src/auth/` | API key validation, permission checks, `ArcSwap`-based auth store with hot-reload |
| `src/client.rs` | `FafbClient` — reusable TCP/TLS client for connecting to a fafb-q server |
| `src/cli/` | CLI command handlers: pub, sub, fetch, info, drain, health, config, bench |
| `src/tls.rs` | `TlsProvider` — rustls wrapper with `ArcSwap` hot reload, mTLS, cert expiry tracking |
| `src/dlq.rs` | Dead-letter queue — `DlqReason`, system header constants, `build_dlq_headers()` |
| `src/wildcard.rs` | `TopicPattern` (NATS-style `*`/`>`), `SubscriptionTrie` for efficient wildcard matching |
| `src/types.rs` | Core types: `MessageEnvelope`, `Header`, `TopicOffset`, `MessageId`, `now_millis()` |
| `src/retention.rs` | Background segment deletion by age/bytes/count, `scan_segments()` shared with compaction |
| `src/error.rs` | `Error` enum, `ErrorCode` wire codes |
| `scripts/install.sh` | One-liner install script — detects Linux platform/arch, downloads latest binary, installs to `/usr/local/bin` |
| `scripts/rolling-upgrade.sh` | Zero-downtime upgrade script — drains running server via admin API, swaps binary, restarts, verifies health |

## Gotchas and Invariants

These are things that look correct but aren't, or constraints not obvious from reading a single file:

- **`MessageId::new(0)` panics** — zero is reserved as a sentinel byte for record format versioning. The v1/v2 record format detection relies on this invariant.
- **`subscribe()` starts at head (latest), not beginning** — use `subscribe_at(offset)` or `subscribe_from(timestamp)` for historical consumption. Tests that publish then subscribe will get zero messages unless they use `subscribe_at`.
- **`MessageEnvelope.timestamp` is `u64` millis since epoch**, not `SystemTime`. Use `now_millis()` from `crate::types`.
- **`publish()` returns `PublishResult { offset, evicted_messages, topic_created }`**, not just an offset. `topic_created` drives wildcard auto-subscribe on the publish hot path.
- **`engine.init()` must complete before TCP traffic** — HTTP starts first (for health probes), TCP accept loop waits on `lifecycle.is_initialized()`.
- **`GoingAway` is NOT an Error frame** — Error frames have `is_fatal()` semantics; GoingAway gives clients a grace period.
- **`Partial` reconnect status means NO active subscriptions** — all successful subscribes are rolled back. Client should fall back to `Connect + SubscribeAt`.
- **`NamespaceManager::subscribe()` returns the group's `next_offset`** (not the requested offset) when joining an existing consumer group.
- **Capability negotiation is bitwise AND** — `agreed = client_caps & SERVER_CAPABILITIES`. Wire format changes (headers, TTL, batch ack) only activate when the corresponding capability bit is negotiated.
- **DLQ messages have `ttl_ms: 0`** — `route_to_dlq()` clears TTL so DLQ messages never expire.
- **`disk_backed` mode** uses disk capacity (`disk_capacity_threshold_pct`) instead of memory budget to gate publishes. This changes the backpressure model entirely.
- **Reserved header prefix `_fafb.`** — rejected on client publish, only server can set these (DLQ headers).
- **Inflight redelivery uses exponential backoff** — `min(base * 2^(count-1), max_backoff) * jitter(±25%)`, exponent capped at 10.
- **`mem_size()` not `payload.len()`** — memory accounting includes header bytes. Use `MessageEnvelope::mem_size()` for budget calculations.
- **`idle_timeout_secs` default is 30, not 300** — clients MUST send periodic Ping frames (recommended every 10s) or the server will close the connection. The `FafbClient` handles this automatically via a background keepalive task. Third-party SDK implementations must implement keepalive with `ping_interval <= idle_timeout / 3`.
- **Topic inactivity GC has a grace period** — topics are not deleted immediately when they exceed `topic_inactive_secs`. They enter a grace period (default 24h) first. Grace period state is in-memory only — lost on restart, requiring full re-qualification.
- **Active subscribers block topic reaping** — a topic with any connected subscriber or inflight messages is never reaped, regardless of activity timestamps.
- **`teardown_topic()` does NOT clean up wildcard subscriptions** — only `TopicDeliveryState` is removed. Wildcard entries in `subscription_trie` / `wildcard_subs` survive so auto-subscribe fires if the topic is re-created.

## Key Design Decisions

**Config hot-reload tiers** — fields are classified by reload safety:
- **Tier 1 (safe)**: Read per-operation. Queue limits, quotas, retention, timeouts, compaction.
- **Tier 2 (coordinated)**: Requires notification. API keys/permissions, namespace add/remove.
- **Tier 3 (restart-only)**: Bind addresses, ports, `data_dir`, `fsync_strategy`, `disk_backed`, `admin_api_key`, `index_interval`.

Runtime config stored in `SharedConfig = Arc<ArcSwap<RuntimeConfig>>` for lock-free reads.

**Crash isolation** — Global panic hook logs through tracing. Background tasks (retention, compaction) are supervised with exponential backoff respawn. Each TCP connection is an independent tokio task. Namespace manager operations return `Result` — no cross-client error propagation.

**Server lifecycle** — `ServerState::Running → Draining → ShuttingDown` (forward-only via `AtomicU8`). Single `watch::channel<ServerState>` broadcasts to all subscribers. There is no separate shutdown bool channel.

**Shutdown sequence** — SIGTERM/SIGINT/HTTP drain → `engine.force_flush()` → `begin_drain()` → accelerate inflight timeouts → wait for connections (or timeout) → `manager.shutdown()` + `engine.shutdown()` → exit.

**Storage format** — Segment files (`.seg`) with sparse index files (`.idx`, 24-byte entries every `index_interval` records). Record v2 format: `[4B len][8B offset][0x00][0x02][bincode(MessageEnvelope)]`. v1 detected by sentinel byte check. Compaction converts v1→v2.

**Protocol capabilities** — Bitmask negotiated at connect: `CAP_GOING_AWAY` (0), `CAP_RECONNECT` (1), `CAP_NACK` (2), `CAP_HEADERS` (3), `CAP_WILDCARD` (4), `CAP_BATCH_ACK` (5), `CAP_MESSAGE_TTL` (6). Features like headers, TTL, and batch ack change wire format only when negotiated.

**Fetch is stateless** — `Fetch`/`FetchFrom` frames and HTTP `GET /v1/{ns}/{topic}/fetch` do not create subscriptions, delivery tasks, or inflight tracking. Clients manage their own offset progression. `Permission::Fetch` is independent of `Permission::Subscribe`.

**Health endpoints** — `GET /health/live` always 200 (liveness), `GET /health/ready` returns 503 when not initialized or draining (readiness). All `/v1/*` routes gated by `require_initialized` middleware.

**Admin endpoints** — `/admin/*` require `Authorization: Bearer <admin_api_key>`. Key endpoints: `POST /admin/drain`, `POST /admin/reload`, `POST /admin/tls/reload`, `GET /admin/config`.

**Topic inactivity GC** — topics auto-created on first publish can be automatically reaped after a configurable inactivity window. Activity = last publish OR last subscribe/fetch (whichever is newest). Info queries do not count. Config is per-namespace with NATS-style wildcard pattern overrides (`topic_inactive_overrides`). Opt-in only (`topic_inactive_secs: None` by default). Soft-delete with 24h grace period before hard delete. Grace period state is in-memory only — on restart, topics re-qualify through the full inactivity window. Admin visibility via `GET /admin/topics/pending-reap`.

## Constructor Patterns

Both the queue engine and namespace manager use config structs with `Default`:

```rust
InMemoryQueueEngine::new(storage, EngineConfig { ..Default::default() })
DefaultNamespaceManager::new(engine, ns_ids, quotas, ManagerConfig { ..Default::default() })
```

## Test Patterns

- Unit tests: inline `#[cfg(test)]` modules in each source file
- Integration tests: `tests/` directory, each file gets its own test binary
- Test helpers: `tests/common/mod.rs` — `TestServer`, `DeployTestServer`, `TestClient`
- `TestClient::connect_and_auth(addr, ns, key)` for basic auth, `connect_and_auth_with_caps` for capability-aware
- Use `subscribe_at_and_consume()` in tests (not `Subscribe`) — `Subscribe` starts at head
- Deploy tests reuse `data_dir` across server instances to simulate restarts
- All metrics prefixed `fafb_*` — discover via `grep -r "fafb_" src/`

## CLI Binary Structure

The `fafb-q` binary is dual-purpose: `fafb-q serve` (or bare `fafb-q --config`) runs the server, other subcommands (`pub`, `sub`, `fetch`, `info`, `drain`, `health`, `config`, `bench`, `completions`) are CLI operations. CLI config at `~/.fafb-q/config.toml` with profile support.
