# Phase 17: Configuration File Hardening & Hot-Reload

## Goal

Harden the existing configuration system with unknown-key rejection, a config dump CLI flag, multi-path file discovery, additional CLI override flags, expanded env var coverage, and enable runtime reconfiguration of the server without restarts. Operators must be able to change queue limits, namespace quotas, retention policies, slow consumer thresholds, and other operational parameters by editing `config.toml` and triggering a reload — without dropping connections, losing messages, or creating inconsistent state.

## Already Implemented (NOT in scope)

The following are **already complete** — do NOT reimplement:

| Capability | Implementation | Location |
|---|---|---|
| Config structs | `ServerConfig`, `TcpConfig`, `HttpConfig`, `StorageConfig`, `QueueConfig`, `DeployConfig`, `LoggingConfig`, `MetricsConfig`, `NamespaceConfig` with `serde::Deserialize` + `Default` | `src/config.rs` |
| CLI args | `CliArgs` with `--config` flag via `clap::Parser` | `src/config.rs:91-98` |
| TOML loading | `ServerConfig::load(path)` reads TOML, applies env overrides, validates | `src/config.rs:718-743` |
| Env var overrides | `apply_env_overrides()` with `FAFB_TCP_ADDRESS`, `FAFB_TCP_PORT`, `FAFB_MAX_MESSAGE_SIZE`, `FAFB_TCP_IDLE_TIMEOUT`, `FAFB_TCP_MAX_CONNECTIONS`, `FAFB_HTTP_ADDRESS`, `FAFB_HTTP_PORT`, `FAFB_DATA_DIR`, `FAFB_ADMIN_API_KEY` | `src/config.rs:651-714` |
| Validation | `ServerConfig::validate()` with port conflict, namespace, key uniqueness, message size, idle timeout, logging, deploy config checks. Collects all errors, not fail-on-first. | `src/config.rs:514-715` |
| ConfigError enum | `PortOutOfRange`, `PortConflict`, `NoNamespaces`, `InvalidNamespaceName`, `ApiKeyTooShort`, `EmptyPermissions`, `DuplicateApiKey`, `MaxMessageSizeTooSmall`, `IdleTimeoutTooSmall`, `InvalidEnvVar`, `InvalidLogging`, `InvalidDeployConfig` | `src/config.rs:442-511` |
| Namespace config | `[namespaces.NAME]` TOML tables with `api_key`, `permissions`, quotas, retention, `max_disk_bytes`, etc. | `src/config.rs:401-431` |
| Example config | `config/config.example.toml` — comprehensive, with all sections documented, multiple namespace examples, env var names in comments | `config/config.example.toml` |
| main.rs integration | `CliArgs::parse()`, `ServerConfig::load(&config_path)`, error handling | `src/main.rs:194-206` |

## Motivation for Hot-Reload

Today, every configuration change requires a full server restart. Even with the graceful deployment protocol (Plan 14), a restart means:

- **Drain latency**: 30+ seconds of GoingAway → drain → shutdown → cold start → engine replay → ready. For large datasets, replay alone can take 10+ seconds.
- **Client disruption**: Every connected client must reconnect, re-authenticate, and re-subscribe. Consumer group state is rebuilt from storage. Inflight messages are redelivered after timeout.
- **Operational friction**: Routine changes — adjusting a namespace quota from 10,000 to 50,000 messages, tightening a retention policy, adding a new namespace — carry the same cost as deploying new code.
- **Incident response**: During a traffic spike, an operator cannot increase `max_connections` or `global_max_memory_bytes` without a full restart cycle.

Config hot-reload eliminates restart-induced disruption for the ~70% of configuration changes that don't affect structural properties (bind addresses, data directories, storage engine mode).

## Prerequisites

- Plans 1–15 complete (all subsystems functional)
- Plan 14 complete (graceful deployment — `ServerLifecycle`, health endpoints, drain protocol)

## Guiding Principles

- **Fail loud, fail early.** Unknown keys, type mismatches, and invalid values cause a startup error with a clear message. No silent fallback to defaults when a key is misspelled.
- **Layered config with clear precedence.** CLI flags > environment variables > config file > compiled defaults.
- **Example file stays in sync.** CI test ensures `config/config.example.toml` parses and validates cleanly.

## Current Gaps

| Gap | Impact | Priority |
|---|---|---|
| No unknown key rejection | Misspelled keys silently ignored — operator thinks they changed `max_mesage_size` but it's a typo | **High** |
| No `--dump-config` flag | Can't verify resolved config after env/CLI override merging | **Medium** |
| No multi-path file discovery | Must always pass `--config`; no `$FAFB_CONFIG` env or `/etc/fafb-q/` fallback | **Medium** |
| Limited CLI override flags | Only `--config` exists; no `--bind`, `--data-dir`, `--log-level` for quick overrides | **Low** |
| Incomplete env var coverage | Missing env vars for many settings (queue limits, storage settings, logging, etc.) | **Medium** |
| No CI test for example config | Example file could drift from struct definitions undetected | **Medium** |
| TOML section names don't match plan proposals | Plan 16 proposes `[engine]` and `[manager]` sections; actual code uses `[queue]` and fields on `TcpConfig`/`NamespaceConfig` | **N/A — plan-only issue** |
| No `server_id` field | No instance identifier for logs/metrics in multi-instance deployments | **Low** |
| No config file permissions check | Config file with API keys could be world-readable | **Low** |
| No runtime config reload | Every config change requires full restart cycle | **High** |

## Config Field Tier Classification

Not all config fields can be changed at runtime. Every field falls into one of three tiers:

| Tier | Can reload? | Reason | Examples |
|------|-------------|--------|----------|
| **1 — Safe** | Yes | Value is read per-operation or per-tick; changing it mid-flight has no structural impact | Queue limits, namespace quotas, retention policies, slow consumer settings, timeouts, compaction settings |
| **2 — Coordinated** | Yes, with care | Requires rebuilding a derived data structure or notifying a subsystem | API keys/permissions (auth store rebuild), adding/removing namespaces (manager state creation/teardown), `max_open_fds` (LRU cache resize) |
| **3 — Restart-only** | No | Changing the value would require re-binding sockets, moving data directories, or fundamentally restructuring engine internals | `bind_address`, `port`, `data_dir`, `fsync_strategy`, `disk_backed`, `admin_api_key`, `index_interval` |

The reload endpoint must **accept** a config file containing Tier 3 changes (they are simply ignored with warnings), so that operators can use the same `config.toml` for both startup and reload without maintaining separate files.

## TOML Section Name Alignment

The plan's proposed TOML structure uses `[engine]` and `[manager]` sections, but the actual code uses `[queue]` and spreads manager settings across `TcpConfig` and `NamespaceConfig`. The example config already reflects the actual structure. **Do not rename sections** — this would be a breaking change for existing users.

| Plan's proposed section | Actual section | Keep as-is? |
|---|---|---|
| `[engine]` | `[queue]` | Yes — matches code |
| `[manager]` | N/A — fields are on `TcpConfig` (slow consumer), `QueueConfig` (cold_batch_size) | Yes — refactoring would break existing configs |
| `[[namespace]]` | `[namespaces.NAME]` | Yes — HashMap-based is simpler and already shipped |
| `[tls]` | Does not exist yet | Plan 16 TLS will add this |

---

## Phase 17.1 — Unknown Key Rejection

### Design

**Problem:** serde's `Deserialize` ignores unknown keys by default. A misspelled `max_messsage_size` is silently dropped — the operator gets the default value and no error.

**Solution:** Add `#[serde(deny_unknown_fields)]` to all config structs:

```rust
#[derive(Debug, Clone, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct TcpConfig { ... }

#[derive(Debug, Clone, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct HttpConfig { ... }

// ... all config structs
```

**Backward compatibility:** This is a breaking change for anyone with typos or extra fields in their config. Log the exact unknown field name in the error message so operators can fix it immediately.

**Note on `NamespaceConfig`:** Currently uses `#[derive(Deserialize)]` without `default` (no `serde(default)` on the struct itself, only on individual fields). Adding `deny_unknown_fields` here is safe — namespace configs have a well-defined schema.

### Tasks

- [x] Add `#[serde(deny_unknown_fields)]` to all config structs: `TcpConfig`, `HttpConfig`, `StorageConfig`, `QueueConfig`, `DeployConfig`, `LoggingConfig`, `MetricsConfig`, `NamespaceConfig`, `RetentionPolicy`
- [x] Verify `config/config.example.toml` still parses cleanly (will catch if the example has any typos)
- [x] Verify all test configs in `src/config.rs` inline tests still parse
- [x] Update error messages to include the unknown field name (serde does this automatically)
- [x] Unit test: config with unknown key `tcp.max_mesage_size` (typo) produces error mentioning the field name

**Verification:** All existing tests pass. Example config parses cleanly. A config with a typo produces a clear, specific error.

---

## Phase 17.2 — Config File Discovery & Permissions

### Design

Extend `main.rs` to search multiple locations when `--config` is not specified:

```rust
fn find_config_path(cli_path: Option<PathBuf>) -> Option<PathBuf> {
    // 1. CLI flag (highest priority)
    if let Some(path) = cli_path {
        return Some(path);
    }

    // 2. FAFB_CONFIG env var
    if let Ok(path) = std::env::var("FAFB_CONFIG") {
        let p = PathBuf::from(path);
        if p.exists() {
            return Some(p);
        }
        tracing::warn!(path = %p.display(), "FAFB_CONFIG points to non-existent file");
    }

    // 3. Current directory
    let local = PathBuf::from("config.toml");
    if local.exists() {
        return Some(local);
    }

    // 4. System-wide
    let system = PathBuf::from("/etc/fafb-q/config.toml");
    if system.exists() {
        return Some(system);
    }

    None // Start with compiled defaults
}
```

When no file is found, start with defaults and log:
```
INFO No config file found (searched: $FAFB_CONFIG, ./config.toml, /etc/fafb-q/config.toml). Using compiled defaults.
```

**Current behavior change:** Today, `main.rs` defaults to `PathBuf::from("config.toml")` and errors if it doesn't exist. The new behavior is more forgiving — no file = use defaults with a log message.

### Config File Permissions Warning

When loading a config file that contains API keys, check file permissions and warn if world-readable:

```rust
#[cfg(unix)]
fn check_config_permissions(path: &Path) {
    use std::os::unix::fs::PermissionsExt;
    if let Ok(meta) = std::fs::metadata(path) {
        let mode = meta.permissions().mode();
        if mode & 0o044 != 0 {
            tracing::warn!(
                path = %path.display(),
                mode = format!("{:o}", mode),
                "config file is world-readable — consider chmod 600 to protect API keys"
            );
        }
    }
}
```

### Tasks

- [x] Implement `find_config_path()` with 4-tier search (CLI → `$FAFB_CONFIG` → `./config.toml` → `/etc/fafb-q/config.toml`)
- [x] Change `main.rs` to use `find_config_path()` instead of hardcoded `PathBuf::from("config.toml")`
- [x] When no file found and `--config` not specified, start with `ServerConfig::load_defaults()` + env overrides + validation
- [x] Add `check_config_permissions()` warning on Unix
- [x] Unit test: discovery order is correct, env var takes precedence over CWD
- [x] Integration test: server starts with no config file (compiled defaults)

**Verification:** Server starts with no config file. `FAFB_CONFIG` env var is respected. World-readable config file produces a warning on Unix.

---

## Phase 17.3 — Config Dump & Additional CLI Flags

### Design

Add `--dump-config` to `CliArgs`:

```rust
#[derive(Debug, Parser)]
#[command(name = "fafb-q", about = "Centralized queue server")]
pub struct CliArgs {
    #[arg(short, long)]
    pub config: Option<PathBuf>,

    /// Print the resolved configuration and exit (secrets redacted)
    #[arg(long)]
    pub dump_config: bool,
}
```

Implementation:
1. Load config normally (file + env + CLI)
2. Serialize to TOML with `toml::to_string_pretty()`
3. Redact API keys: replace all but last 4 chars with `****`
4. Print to stdout and exit(0)

Requires adding `serde::Serialize` to all config structs (currently only `Deserialize`).

### Additional CLI Override Flags

Add commonly-used override flags to `CliArgs` for container/script use:

```rust
pub struct CliArgs {
    #[arg(short, long)]
    pub config: Option<PathBuf>,

    #[arg(long)]
    pub dump_config: bool,

    /// Override TCP bind address (e.g., "0.0.0.0:9876")
    #[arg(long)]
    pub tcp_bind: Option<String>,

    /// Override HTTP bind address (e.g., "0.0.0.0:8080")
    #[arg(long)]
    pub http_bind: Option<String>,

    /// Override storage data directory
    #[arg(long)]
    pub data_dir: Option<String>,

    /// Override log level
    #[arg(long)]
    pub log_level: Option<String>,
}
```

CLI flags are applied AFTER env var overrides in `ServerConfig::load()`.

### Tasks

- [x] Add `serde::Serialize` derive to all config structs
- [x] Add `--dump-config` flag to `CliArgs`
- [x] Implement config dump with API key redaction
- [x] Add `--tcp-bind`, `--http-bind`, `--data-dir`, `--log-level` CLI flags
- [x] Apply CLI overrides after env overrides via `ServerConfig::apply_cli_overrides()`
- [x] Unit test: CLI flags override env vars and file values
- [x] Snapshot test: `--dump-config` output roundtrips as valid TOML

**Verification:** `--dump-config` prints resolved config with redacted API keys and exits. CLI flags override env vars override file values. `cargo test` passes.

---

## Phase 17.4 — Expanded Env Vars & CI Tests

### Design

Current coverage is limited to TCP/HTTP bind addresses, ports, data dir, and admin API key. Add env vars for commonly-tuned settings:

| Config Path | Env Var | Currently Implemented? |
|---|---|---|
| `tcp.bind_address` | `FAFB_TCP_ADDRESS` | Yes |
| `tcp.port` | `FAFB_TCP_PORT` | Yes |
| `tcp.max_message_size` | `FAFB_MAX_MESSAGE_SIZE` | Yes |
| `tcp.idle_timeout_secs` | `FAFB_TCP_IDLE_TIMEOUT` | Yes |
| `tcp.max_connections` | `FAFB_TCP_MAX_CONNECTIONS` | Yes |
| `http.bind_address` | `FAFB_HTTP_ADDRESS` | Yes |
| `http.port` | `FAFB_HTTP_PORT` | Yes |
| `storage.data_dir` | `FAFB_DATA_DIR` | Yes |
| `deploy.admin_api_key` | `FAFB_ADMIN_API_KEY` | Yes |
| `queue.global_max_memory_bytes` | `FAFB_GLOBAL_MAX_MEMORY_BYTES` | **No** |
| `queue.max_depth` | `FAFB_MAX_DEPTH` | **No** |
| `queue.disk_backed` | `FAFB_DISK_BACKED` | **No** |
| `queue.disk_capacity_threshold_pct` | `FAFB_DISK_CAPACITY_THRESHOLD_PCT` | **No** |
| `logging.level` | `FAFB_LOG_LEVEL` (+ existing `RUST_LOG`) | **No** (only `RUST_LOG` at subscriber level) |
| `logging.format` | `FAFB_LOG_FORMAT` | **No** |
| `metrics.enabled` | `FAFB_METRICS_ENABLED` | **No** |
| `storage.max_open_fds` | `FAFB_MAX_OPEN_FDS` | **No** |

> **Note:** Not every field needs an env var. Focus on settings commonly tuned in container environments (memory limits, bind addresses, log levels, data paths).

### Tasks

- [x] Add missing env var overrides to `apply_env_overrides()`: `FAFB_GLOBAL_MAX_MEMORY_BYTES`, `FAFB_MAX_DEPTH`, `FAFB_DISK_BACKED`, `FAFB_DISK_CAPACITY_THRESHOLD_PCT`, `FAFB_LOG_LEVEL`, `FAFB_LOG_FORMAT`, `FAFB_METRICS_ENABLED`, `FAFB_MAX_OPEN_FDS`
- [x] Add CI test: load `config/config.example.toml`, verify parse + validation passes
- [x] Add CI test: load `config/config.sdk-test.toml`, verify parse + validation passes
- [x] Update `config/config.example.toml` with any new env var names in comments
- [x] Unit tests for each new env var override
- [x] Add `FAFB_BACKPRESSURE_POLICY` env var with `parse_backpressure_policy()` helper (accepts "Reject"/"DropOldest"/"Block")
- [x] Validate `FAFB_LOG_FORMAT` immediately in `apply_env_overrides()` (rejects invalid values at assignment, not deferred to `validate()`)
- [x] Add `parse_bool()` helper for bool env vars (accepts "true"/"false"/"1"/"0", case-insensitive)
- [x] Fix misleading `cold_batch_size` comment in `config.example.toml` — field is on `ManagerConfig`, not `QueueConfig`; uncommenting would fail with `deny_unknown_fields`
- [x] Add multi-env-var interaction tests (disk-backed pair, 5-var combined test)

**Verification:** CI tests validate both example configs. All new env vars are documented in `config.example.toml`. 87 config tests, 478 total library tests pass. No new clippy warnings.

---

## Phase 17.5 — SharedConfig Infrastructure & ArcSwap Integration

### Goal

Introduce the `RuntimeConfig` struct, the `ArcSwap`-based shared config store, and thread it through `main.rs` — without changing any subsystem behavior. After this phase, the shared config exists and is populated at startup, but no subsystem reads from it yet (they still use their owned copies).

### Why ArcSwap

The [`arc-swap`](https://docs.rs/arc-swap) crate provides lock-free reads with atomic pointer swaps:

- **Lock-free reads on the hot path**: `ArcSwap::load()` is ~2ns — comparable to an `AtomicU64` load. Every publish, subscribe, and deliver operation reads config; contention from `RwLock` would be measurable at scale.
- **Snapshot consistency**: A single `store()` swaps all related fields atomically. Readers see either the old config or the new config, never a partial mix.
- **No reader starvation**: Unlike `RwLock`, writers never block readers. The old `Arc` is dropped only after all readers release it (epoch-based reclamation).
- **Zero transitive dependencies**: `arc-swap` has no deps.

### Design

#### RuntimeConfig struct

A new struct containing only the Tier 1 and Tier 2 fields — the subset of `ServerConfig` that can change at runtime. This struct is the unit of atomic swap.

```rust
use arc_swap::ArcSwap;
use std::sync::Arc;

/// Configuration fields that can be changed at runtime via hot-reload.
/// Stored in an ArcSwap for lock-free reads. Fields that require a restart
/// (bind addresses, data directory, fsync strategy, disk_backed mode) are
/// intentionally excluded.
#[derive(Debug, Clone)]
pub struct RuntimeConfig {
    // --- Queue engine (Tier 1) ---
    pub max_depth: usize,
    pub default_topic_max_bytes: u64,
    pub backpressure_policy: BackpressurePolicy,
    pub global_max_memory_bytes: u64,
    pub max_consumer_groups_per_topic: usize,
    pub disk_capacity_threshold_pct: f64,
    pub max_redelivery_attempts: u32,
    pub dlq_topic_suffix: String,

    // --- TCP / connection behavior (Tier 1) ---
    pub idle_timeout_secs: u64,
    pub max_connections: usize,
    pub subscriber_channel_buffer: usize,
    pub slow_consumer_policy: SlowConsumerPolicy,
    pub slow_consumer_threshold: u64,
    pub slow_consumer_disconnect_after: u64,
    pub max_message_size: u32,

    // --- HTTP (Tier 1) ---
    pub max_fetch_response_bytes: usize,

    // --- Storage behavior (Tier 1) ---
    pub max_segments_per_topic: usize,
    pub retention_check_interval_secs: u64,
    pub segment_merge_threshold_bytes: u64,
    pub compaction_interval_secs: u64,
    pub batch_flush_interval_ms: u64,
    pub batch_flush_size_bytes: u64,
    pub max_open_fds: usize,

    // --- Deploy (Tier 1) ---
    pub drain_timeout_secs: u64,
    pub shutdown_timeout_secs: u64,
    pub going_away_reason: String,

    // --- Per-namespace (Tier 1 + Tier 2) ---
    pub namespace_quotas: HashMap<NamespaceId, NamespaceQuota>,
    pub namespace_retention: HashMap<NamespaceId, RetentionPolicy>,

    // --- Auth (Tier 2 — requires auth store rebuild) ---
    pub namespace_auth: HashMap<NamespaceId, NamespaceAuthConfig>,
}

/// Auth-relevant fields extracted from NamespaceConfig.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct NamespaceAuthConfig {
    pub api_key: ApiKey,
    pub permissions: Vec<Permission>,
}

/// Shared, atomically-swappable config handle. Clone is cheap (Arc clone).
pub type SharedConfig = Arc<ArcSwap<RuntimeConfig>>;

pub fn shared_config(config: &ServerConfig) -> SharedConfig {
    Arc::new(ArcSwap::from_pointee(RuntimeConfig::from_server_config(config)))
}
```

> **Why a separate struct instead of reusing `ServerConfig`?** `ServerConfig` contains `serde(Deserialize)` attributes, TOML-specific default functions, and Tier 3 fields. `RuntimeConfig` is a runtime-oriented view — no serde, no defaults, no restart-only fields. Keeping them separate makes the Tier boundary explicit in the type system.

#### Conversion from ServerConfig to RuntimeConfig

```rust
impl RuntimeConfig {
    pub fn from_server_config(config: &ServerConfig) -> Self {
        let mut namespace_quotas = HashMap::new();
        let mut namespace_retention = HashMap::new();
        let mut namespace_auth = HashMap::new();

        for (ns_id, ns_cfg) in &config.namespaces {
            namespace_quotas.insert(ns_id.clone(), NamespaceQuota { /* ... */ });
            namespace_retention.insert(ns_id.clone(), ns_cfg.retention.clone());
            namespace_auth.insert(ns_id.clone(), NamespaceAuthConfig {
                api_key: ns_cfg.api_key.clone(),
                permissions: ns_cfg.permissions.clone(),
            });
        }

        RuntimeConfig { /* all Tier 1 + 2 fields mapped from config */ }
    }
}
```

#### main.rs integration

After `ServerConfig::load()` succeeds, create the shared config:

```rust
let shared_config = config::shared_config(&config);
```

Add `shared_config: SharedConfig` and `config_path: Arc<PathBuf>` to `AppState`.

### Tasks

- [x] Add `arc-swap = "1"` to `Cargo.toml`
- [x] Define `RuntimeConfig` struct with all Tier 1 and Tier 2 fields
- [x] Define `NamespaceAuthConfig` struct
- [x] Implement `RuntimeConfig::from_server_config()`
- [x] Define `SharedConfig` type alias and `shared_config()` constructor
- [x] Initialize `SharedConfig` in `main.rs` after config load
- [x] Add `shared_config: SharedConfig` to `AppState`
- [x] Add `config_path: Option<Arc<PathBuf>>` to `AppState` (`None` when running on compiled defaults — reload endpoint can distinguish)
- [x] Write unit tests for `RuntimeConfig` conversion and `ArcSwap` behavior
- [x] Verify `cargo test` passes — no behavioral changes, just new code
- [x] Add `Default` impl for `ServerConfig` (simplifies test construction and `load_defaults()`)
- [x] Move `NamespaceQuota` from `namespace/manager.rs` to `config.rs` (natural home alongside `NamespaceConfig`; `pub use` re-export preserves all existing import paths)
- [x] Add `Hash` derive to `NamespaceAuthConfig` (enables efficient change detection in Phase 17.6 diffing)
- [x] Add `runtime_config_exhaustiveness_guard` test — destructures all `RuntimeConfig` fields so the compiler catches unmapped fields

**Verification:** `SharedConfig` is created in `main.rs` and threaded into `AppState`. No subsystem reads from it yet. All existing tests pass unchanged. 492 lib tests + 79 integration tests pass. `config_path` is `Option` so reload endpoint can reject when no file was loaded.

---

## Phase 17.6 — Admin Reload Endpoint & Config Diffing

### Goal

Add `POST /admin/reload` that re-reads `config.toml`, validates the new config, diffs it against the current `RuntimeConfig`, rejects invalid changes, applies valid changes via `ArcSwap::store()`, and returns a detailed JSON response. Also add a `watch::channel` notification mechanism so subsystems that need active reaction (auth store rebuild, namespace creation) can subscribe.

### Design

#### Config diff engine

```rust
#[derive(Debug, Clone, serde::Serialize)]
pub struct ConfigChange {
    pub field: String,
    pub description: String,
    pub requires_notification: bool,
}

#[derive(Debug, Clone, serde::Serialize)]
pub struct ConfigDiff {
    pub applied: Vec<ConfigChange>,
    pub ignored: Vec<ConfigChange>,
    pub namespaces_added: Vec<NamespaceId>,
    pub namespaces_removed: Vec<NamespaceId>,
}

impl RuntimeConfig {
    pub fn diff(&self, old: &RuntimeConfig) -> ConfigDiff { ... }
}
```

The diff function compares each field explicitly (not via derive macro) to ensure compiler warnings catch new `RuntimeConfig` fields that aren't covered.

#### Tier 3 change detection

```rust
pub fn detect_restart_only_changes(old: &ServerConfig, new: &ServerConfig) -> Vec<ConfigChange> {
    // Compares: tcp.bind_address, tcp.port, http.bind_address, http.port,
    // storage.data_dir, storage.fsync_strategy, queue.disk_backed, deploy.admin_api_key
}
```

#### Reload notification channel

Tier 2 changes (auth store rebuild, namespace creation/removal) require active notification:

```rust
#[derive(Debug, Clone)]
pub struct ReloadNotification {
    pub auth_changed: bool,
    pub namespaces_added: Vec<NamespaceId>,
    pub namespaces_removed: Vec<NamespaceId>,
}
```

Uses `tokio::sync::watch::Sender<Option<ReloadNotification>>` initialized to `None`.

#### Concurrency control

Both HTTP endpoint and file watcher acquire a `tokio::sync::Mutex` before proceeding to prevent concurrent reloads:

```rust
pub struct ReloadState {
    lock: Mutex<()>,
    config_path: Arc<PathBuf>,
    shared_config: SharedConfig,
    current_server_config: RwLock<ServerConfig>,
    reload_notify: watch::Sender<Option<ReloadNotification>>,
}
```

#### Core reload function

```rust
pub fn reload_config(
    config_path: &Path,
    shared_config: &SharedConfig,
    current_server_config: &RwLock<ServerConfig>,
    lifecycle: &ServerLifecycle,
    reload_notify: &watch::Sender<Option<ReloadNotification>>,
) -> Result<ConfigDiff, ReloadError> {
    // 1. Read + parse + validate
    // 2. Detect Tier 3 changes (logged as warnings, not applied)
    // 3. Build new RuntimeConfig and diff
    // 4. Check namespace removal safety
    // 5. Apply via ArcSwap::store() if anything changed
    // 6. Send Tier 2 notifications
}
```

#### HTTP endpoint

`POST /admin/reload` — same admin auth as `/admin/drain`. Returns JSON response with diff details. Route registered alongside `/admin/drain` in admin router.

Response codes:
- 200 `{ "status": "applied", "changes_applied": 3, ... }` — changes applied
- 200 `{ "status": "unchanged" }` — no changes detected
- 400 `{ "status": "error", "message": "..." }` — validation error
- 409 `{ "status": "error", "message": "..." }` — namespace removal blocked

### Reload is not atomic with file save

An editor saving `config.toml` may write a truncated file momentarily. The reload function reads → parses → validates as one operation. Partial TOML fails to parse; old config is retained.

### Tasks

- [x] Define `ConfigChange`, `ConfigDiff` structs with serde Serialize
- [x] Implement `RuntimeConfig::diff()` with per-field comparison
- [x] Implement `detect_restart_only_changes()` for Tier 3 fields
- [x] Define `ReloadNotification` struct
- [x] Define `ReloadError` enum
- [x] Implement `ReloadState` struct with `tokio::sync::Mutex` to serialize concurrent reload calls
- [x] Implement `reload_config()` core function (acquires `ReloadState.lock`)
- [x] Add `shared_config`, `config_path`, `current_server_config`, `reload_notify`, `reload_state` to `AppState`
- [x] Implement `POST /admin/reload` handler with admin auth
- [x] Register `/admin/reload` route in admin router
- [x] Wire `reload_notify` watch channel in `main.rs`
- [x] Write config diff unit tests (exhaustiveness, per-field, edge cases)
- [x] Write Tier 3 detection unit tests
- [x] Write reload function unit tests (valid, invalid, namespace removal)
- [x] Write HTTP integration tests for the reload endpoint
- [x] Verify `cargo test` passes

**Verification:** `POST /admin/reload` returns detailed diff JSON. Invalid configs are rejected cleanly. Tier 3 changes are logged as warnings but not applied. Concurrent reloads are serialized. 516 lib tests + 7 HTTP reload integration tests pass. `PartialEq` added to `RetentionPolicy` and `NamespaceQuota` for diffing. `ConfigDiff.auth_changed` tracks auth changes directly (no string matching). `ReloadState` stores `CliArgs` to preserve CLI override precedence on reload. Uses `tokio::fs::read_to_string` for async file I/O. Namespace removal logged with warning; full safety check deferred to Phase 17.7.

---

## Phase 17.7 — Subsystem Migration to SharedConfig

### Goal

Migrate subsystems from owned config copies to `SharedConfig` reads, one at a time, starting with the lowest-risk consumers and ending with the hot-path. After this phase, config changes applied via `POST /admin/reload` take effect in real time across all subsystems.

### Migration Strategy

Each subsystem is migrated independently. The pattern is:

1. Add `shared_config: SharedConfig` to the subsystem's constructor / struct
2. Replace reads of owned config fields with `shared_config.load().field_name`
3. For fields read in tight loops, load the config once at the start of the operation (not per-iteration)
4. Remove the now-unused owned config fields
5. Verify tests pass

> **Performance rule:** `shared_config.load()` returns a `Guard` that derefs to `Arc<RuntimeConfig>`. For operations that read multiple config fields, call `.load()` once and bind the result. For operations spanning multiple `.await` points, use `.load_full()` which returns an `Arc<RuntimeConfig>`.

### Per-connection config behavior

- **New connections pick up new config**: The connection handler reads `SharedConfig` when the connection is established.
- **Existing connections re-read selectively**: Only for settings where the operational benefit is clear (e.g., slow consumer thresholds, idle timeout).

### 17.7.1: Background tasks (retention, compaction, cleanup) — Risk: Very low

Read config once per tick (every 60–300 seconds). Replace owned config fields with `SharedConfig` reads.

- [x] Migrate retention task to read `SharedConfig` per tick — remove owned `namespace_configs` field
- [x] Migrate compaction task to read `SharedConfig` per tick — remove captured config values
- [x] Migrate cleanup task to read `SharedConfig` per tick — remove captured `max_segments`
- [x] Verify all background task tests pass

### 17.7.2: Namespace manager quotas — Risk: Low

Replace owned `quotas: HashMap<NamespaceId, NamespaceQuota>` with `SharedConfig` reads on quota check paths.

- [x] Add `SharedConfig` to `DefaultNamespaceManager` constructor
- [x] Migrate namespace quota checks to read `SharedConfig` — remove owned `quotas` field
- [x] Write integration test: change quota via reload, verify new quota enforced immediately

### 17.7.3: Manager config (delivery settings) — Risk: Low-medium

Migrate `max_redelivery_attempts`, `slow_consumer_threshold`, `slow_consumer_disconnect_after`, `cold_batch_size`, `max_consumer_groups_per_topic` to read from `SharedConfig`. Note that `delivery_channel_capacity` cannot change for existing channels (the mpsc channel is already allocated) but new subscriptions pick up the new value.

- [x] Migrate manager delivery settings to read `SharedConfig` — remove owned `ManagerConfig` fields where possible
- [x] Document that `delivery_channel_capacity` applies to new subscriptions only
- [x] Write integration test: change retention policy via reload, verify next sweep uses new policy

### 17.7.4: Auth store (Tier 2 — notification-driven rebuild) — Risk: Medium

Change `ApiKeyStore.entries` from `Arc<[KeyEntry]>` to `ArcSwap<Vec<KeyEntry>>`. Implement `spawn_reload_watcher()` that subscribes to `reload_notify` and rebuilds the entries array when `auth_changed == true`.

> **Security consideration:** Changing an API key invalidates the old key immediately for new connections. Existing connections remain authenticated (the session stores the validated namespace, not the key).

- [x] Change `ApiKeyStore.entries` from `Arc<[KeyEntry]>` to `ArcSwap<Vec<KeyEntry>>`
- [x] Implement `ApiKeyStore::build_entries_from_runtime_config()`
- [x] Implement `ApiKeyStore::spawn_reload_watcher()` for auth store rebuild on notification
- [x] Write integration test: change API key via reload, verify old key rejected on new connections

### 17.7.5: TCP connection handler settings — Risk: Low

New connections pick up new config. For `idle_timeout_secs`, re-read on each idle check so timeout changes affect existing connections. For `max_message_size`, the frame codec is initialized once per connection — new connections only.

- [x] Add `SharedConfig` to `TcpServer` and `ConnectionConfig`
- [x] Migrate idle timeout to re-read `SharedConfig` dynamically on every frame/delivery in `run_connection` — existing connections pick up timeout changes immediately
- [x] Document that `max_message_size` applies to new connections only (existing connections keep old value)

### 17.7.6: Queue engine limits — Risk: Medium (publish hot path)

Read from `SharedConfig` on publish. Since `publish()` already does multiple checks, one additional `Arc` load is negligible.

> **Benchmark requirement:** Run existing publish benchmarks before and after. Expected overhead: <1%. If measurable, cache config per-batch instead of per-publish.

- [x] Add `SharedConfig` to `InMemoryQueueEngine` via `with_shared_config()` constructor
- [x] Migrate publish-path limit checks to read `SharedConfig`
- [x] Run publish benchmarks before and after engine migration — verify <1% overhead

### 17.7.7: HTTP server settings — Risk: Very low

Fetch handler reads `max_fetch_response_bytes` from `SharedConfig`. Remove `MaxFetchResponseBytes` newtype and its `Extension` injection.

- [x] Migrate HTTP fetch handler to read `max_fetch_response_bytes` from `SharedConfig`
- [x] Remove `MaxFetchResponseBytes` newtype struct and Extension injection entirely — replaced with plain `usize` local

### 17.7.8: Namespace addition/removal via reload

Subscribe to `reload_notify` and create namespace state for newly added namespaces. Implement `register_namespace()` (idempotent — if namespace already exists, update quota instead of erroring). Reject removals if active connections exist.

- [x] Implement `DefaultNamespaceManager::register_namespace()` (idempotent)
- [x] Implement `DefaultNamespaceManager::spawn_reload_watcher()` for namespace addition
- [x] Implement `DefaultNamespaceManager::active_connections_for_namespace()` for removal safety checks
- [x] Wire removal safety check into `reload_config()` via `NamespaceConnectionChecker` callback (`OnceLock`-based, set once at startup)
- [x] Write integration test: add namespace via reload, verify publish/subscribe works on new namespace

### Cross-cutting tasks

- [x] Update all affected unit tests to construct subsystems with `SharedConfig`
- [x] Verify `cargo test` and `cargo clippy` pass
- [x] Verify publish benchmark results acceptable
- [x] Slim `ManagerConfig` to only `delivery_channel_capacity` — removed 7 dead fields (`max_redelivery_attempts`, `dlq_topic_suffix`, `max_consumer_groups_per_topic`, `slow_consumer_policy/threshold/disconnect_after`, `cold_batch_size`)
- [x] Add `disk_capacity_threshold_pct` clamping (`.clamp(1.0, 99.9)`) in engine publish path to guard against invalid values from hot-reload

**Verification:** All subsystems read from `SharedConfig`. Config changes via `POST /admin/reload` take effect in real time across all subsystems. 512 lib + 149 integration tests pass. 4 pre-existing clippy warnings, no new ones.

Key implementation details:
- `MaxFetchResponseBytes` newtype removed entirely — fetch handler reads `max_fetch_response_bytes` as plain `usize` from `SharedConfig`.
- Auth store uses `ArcSwap<Vec<KeyEntry>>` with `spawn_reload_watcher()` for notification-driven rebuild.
- Namespace removal blocked when active connections exist via `NamespaceConnectionChecker` callback in `OnceLock`.
- `cold_batch_size` added to `RuntimeConfig` for delivery task hot-reload.
- Engine `disk_capacity_threshold_pct` startup clamping removed; inline `.clamp(1.0, 99.9)` added to publish read path.
- `ManagerConfig` slimmed to single field `delivery_channel_capacity` (mpsc channels are fixed-size at creation).
- `ConnectionConfig` holds `SharedConfig`; `run_connection` re-reads `idle_timeout_secs` dynamically on every frame/delivery — existing connections pick up timeout changes immediately.
- `DeliveryTaskContext` holds `SharedConfig` instead of 6 individual config fields; delivery tasks load config once per loop iteration.
- `InMemoryQueueEngine::with_shared_config()` constructor for production; `new()` convenience builds a fallback `SharedConfig` from `EngineConfig` values for backward-compatible test usage.

---

## Phase 17.8 — File Watcher & Observability

### Goal

Add optional automatic config reload on file change (no HTTP call needed), Prometheus metrics for reload observability, and a `GET /admin/config` introspection endpoint.

### 17.8.1: File watcher with debounce

Add `notify` crate dependency:

```toml
notify = { version = "7", default-features = false, features = ["macos_kqueue"] }
```

Implement `spawn_config_watcher()` with 2-second debounce. Watch the parent directory (not just the file) to handle:
1. Kubernetes ConfigMap mounts (atomic symlink swap to new directory)
2. Editors that write to a temp file then rename (vim, emacs)
3. Direct file modification

### 17.8.2: Hot-reload config section

```rust
#[derive(Debug, Clone, Deserialize)]
pub struct HotReloadConfig {
    #[serde(default)]
    pub watch_file: bool,

    #[serde(default = "default_debounce_secs")]
    pub debounce_secs: u64,
}
```

Add `[hot_reload]` to `ServerConfig`. `hot_reload` itself is a Tier 3 field — you can't hot-reload the hot-reload config.

### 17.8.3: Prometheus metrics

```
fafb_config_reloads_total{trigger, result}      — counter: reload attempts (trigger=http|file_watcher, result=success|error|unchanged)
fafb_config_reload_duration_seconds              — histogram: time from file read to ArcSwap store
fafb_config_last_reload_timestamp                — gauge: Unix timestamp of last successful reload
fafb_config_last_reload_changes                  — gauge: changes applied in last reload
```

### 17.8.4: Config introspection endpoint

`GET /admin/config` — returns the current `RuntimeConfig` as JSON with API keys redacted to `****<last4>`. Requires admin auth.

### Interaction with Plan 16 (TLS)

If Plan 16 TLS is implemented before this plan, `RuntimeConfig` should include cert file paths as Tier 2 fields with `TlsProvider` subscribing to reload notifications. If this plan lands first, add placeholder comments for TLS fields.

### Interaction with MetricsConfig and LoggingConfig

- **`MetricsConfig`** — Tier 3 (restart-only). The Prometheus exporter is initialized once at startup.
- **`LoggingConfig.level`** — Already hot-reloadable via `PUT /admin/log-level`. Config hot-reload should detect level changes and forward them to the existing log level reload handle.
- **Other `LoggingConfig` fields** — Tier 3 (restart-only).

### Tasks

- [x] Add `notify = "7"` to `Cargo.toml`
- [x] Define `HotReloadConfig` struct with `watch_file` and `debounce_secs`
- [x] Add `[hot_reload]` to `ServerConfig` with `Default` implementation
- [x] Implement `spawn_config_watcher()` with debounced file watching
- [x] Wire conditional watcher startup in `main.rs`
- [x] Add `fafb_config_reloads_total` counter metric
- [x] Add `fafb_config_reload_duration_seconds` histogram metric
- [x] Add `fafb_config_last_reload_timestamp` gauge metric
- [x] Add `fafb_config_last_reload_changes` gauge metric
- [x] Instrument `reload_config()` and HTTP handler with metrics
- [x] Implement `GET /admin/config` introspection endpoint with API key redaction
- [x] Register `/admin/config` route in admin router
- [x] Update `config/config.example.toml` with `[hot_reload]` section
- [x] Write file watcher unit tests (debounce, invalid config survival, file delete/recreate, symlink swap)
- [x] Write metrics integration tests
- [x] Write config introspection integration tests
- [x] Write file watcher end-to-end integration test
- [x] Write reload-during-drain rejection test
- [x] Verify `cargo test` and `cargo clippy` pass
- [x] Update CLAUDE.md with hot-reload documentation (config field tiers, admin endpoints, metrics)

**Verification:** File watcher detects config changes with debounce. Metrics record reload events. `GET /admin/config` returns current runtime config with redacted secrets. Invalid config file changes are survived gracefully. Reload rejected during drain with 503. 520 lib tests + 33 HTTP integration tests + 675 total pass. 4 pre-existing clippy warnings, no new ones.

Key implementation details:
- `notify = { version = "7", default-features = false, features = ["macos_kqueue"] }` avoids C dependency for macOS fsevent.
- `spawn_config_watcher()` watches both the parent directory (Kubernetes ConfigMap symlink swaps, editor rename patterns) AND the config file directly (required on macOS — kqueue directory watches don't fire for content changes to existing files).
- Debounce via `tokio::time::sleep()` + drain — coalesces rapid writes into a single reload. After debounce, mtime stability check waits up to 3x500ms if the file was modified within the last 500ms (prevents reading partially-written files).
- Watcher failure is non-fatal — logs error, HTTP reload always works as fallback.
- `HotReloadConfig` is Tier 3 (restart-only) — added to `detect_restart_only_changes()`.
- `RuntimeConfig` has `Serialize` derive; `to_json_redacted()` uses structural `RedactedRuntimeConfig` view type (not fragile JSON path walking) — redacts API keys via `redact_secret()` at the struct level before serialization.
- `NamespaceQuota` and `NamespaceAuthConfig` gained `Serialize` derive for `RuntimeConfig` serialization.
- Config watcher crash-isolated in `main.rs` monitoring loop with exponential backoff respawn. `option_join()` helper cleanly handles the `Option<JoinHandle>` in the `select!` loop.
- Both HTTP and file watcher paths record `fafb_config_reloads_total{trigger,result}`, `fafb_config_reload_duration_seconds`, `fafb_config_last_reload_timestamp`, `fafb_config_last_reload_changes`.
- `ReloadState` holds `Option<Arc<ServerLifecycle>>` — `reload()` rejects with `ReloadError::ServerDraining` (HTTP 503) when server is draining. `ReloadState` is created after lifecycle in `main.rs` so drain-awareness is wired from the start.
- E2E file watcher tests verify: (1) config change detected and applied after debounce, (2) invalid TOML survives without overwriting previous config, (3) watcher recovers and applies subsequent valid changes.

---

## Dependency Graph

```
Phase 17.1 (Unknown key rejection)
Phase 17.2 (Config file discovery)        ──► can be done in parallel
Phase 17.3 (Config dump + CLI flags)
Phase 17.4 (Expanded env vars + CI tests)
                │
                ▼
Phase 17.5 (SharedConfig + ArcSwap) ──► Phase 17.6 (Admin reload endpoint + diff)
                                              │
                                              ├──► Phase 17.7 (Subsystem migration)
                                              │         │
                                              │         ▼
                                              └──► Phase 17.8 (File watcher + observability)
```

**Recommended order**: 17.1–17.4 can be done in parallel or any order. 17.5 → 17.6 → 17.7 → 17.8 must be linear.

---

## Dependencies

```toml
# Already present:
toml = "0.8"
clap = { version = "4", features = ["derive", "env"] }
serde = { version = "1", features = ["derive"] }

# New for hot-reload:
arc-swap = "1"    # Phase 17.5 — zero transitive deps
notify = "7"      # Phase 17.8 — optional file watcher
```

Adding `Serialize` to config structs uses `serde` already in the tree.

---

## Testing Strategy

- **Unit tests:** Unknown key rejection (typos produce clear errors), config discovery path ordering, CLI flag precedence over env vars, env var precedence over file, new env var overrides parse correctly, `RuntimeConfig` conversion, `ArcSwap` behavior, config diff (exhaustiveness, per-field, edge cases), Tier 3 detection, reload function (valid, invalid, namespace removal)
- **Integration tests:** Server starts with no config file (defaults), server starts with example config, broken config produces clear multi-error output, reload via HTTP endpoint, reload with invalid config, reload updates live config, change quota via reload, add namespace via reload, change API key via reload
- **CI:** `config/config.example.toml` and `config/config.sdk-test.toml` are loaded and validated every build
- **Snapshot tests:** `--dump-config` output matches expected resolved config with redacted secrets
- **File watcher tests:** Debounce, invalid config survival, file delete/recreate, symlink swap, end-to-end
- **Metrics tests:** Reload metrics recorded after HTTP trigger, config introspection endpoint

---

## Risks and Mitigations

| Risk | Mitigation |
|---|---|
| `deny_unknown_fields` breaks existing user configs with extra/typo fields | Intentional — the whole point is to catch typos. The error message will name the offending field. Document in release notes. |
| Config file discovery changes default behavior | Today, missing `config.toml` is a fatal error. New behavior (use defaults) is less strict. Gate behind a deprecation warning. |
| `Serialize` derive on config structs increases compile time | Negligible — serde `Serialize` is lightweight. |
| CLI flag proliferation | Keep minimal — only the 4 most commonly overridden settings. Everything else uses env vars or the config file. |
| ArcSwap overhead on publish hot path | Benchmark before/after. `ArcSwap::load()` is ~2ns. If measurable, cache config snapshot per-batch. |
| Concurrent reload calls (HTTP + file watcher) | `ReloadState` with `tokio::sync::Mutex` serializes reload attempts. No inconsistent state. |
| Config file written partially (editor mid-save) | Reload reads → parses → validates as one operation. Partial TOML fails to parse; old config retained. File watcher debounce (2s) helps. |
| Auth store rebuild drops in-progress linear scan | `ArcSwap` on `entries` means old `Arc` stays alive until all readers release it. No torn reads. |
| Namespace quota decrease causes immediate rejections | Intentional — existing messages not deleted. Document in admin endpoint response. |
| `max_connections` decrease with existing connections over new limit | New connections rejected; existing NOT forcibly closed. Count drops naturally. |
| `notify` crate platform differences | `macos_kqueue` on macOS, `inotify` on Linux. Fall back to polling if platform watcher fails. Watcher failure is non-fatal — HTTP reload always works. |
| Diff function falls behind RuntimeConfig fields | `diff_exhaustiveness` unit test catches newly added fields. Compiler warns on unused fields. |

---

## Appendix: Complete Tier Classification

### Tier 1 — Safe to hot-reload (no structural impact)

| Section | Field | Read frequency |
|---------|-------|----------------|
| `[queue]` | `max_depth` | Per publish |
| `[queue]` | `default_topic_max_bytes` | Per publish |
| `[queue]` | `backpressure_policy` | Per publish (at limit) |
| `[queue]` | `global_max_memory_bytes` | Per publish |
| `[queue]` | `max_consumer_groups_per_topic` | Per subscribe |
| `[queue]` | `disk_capacity_threshold_pct` | Per publish (disk_backed only) |
| `[tcp]` | `idle_timeout_secs` | Per idle check (~1/s per connection) |
| `[tcp]` | `max_connections` | Per accept |
| `[tcp]` | `subscriber_channel_buffer` | Per new subscription |
| `[tcp]` | `slow_consumer_policy` | Per new subscription |
| `[tcp]` | `slow_consumer_threshold` | Per slow consumer check |
| `[tcp]` | `slow_consumer_disconnect_after` | Per slow consumer check |
| `[tcp]` | `max_message_size` | Per new connection (codec init) |
| `[http]` | `max_fetch_response_bytes` | Per fetch request |
| `[storage]` | `max_segments_per_topic` | Per cleanup tick |
| `[storage]` | `retention_check_interval_secs` | Per retention tick |
| `[storage]` | `segment_merge_threshold_bytes` | Per compaction tick |
| `[storage]` | `compaction_interval_secs` | Per compaction tick |
| `[storage]` | `batch_flush_interval_ms` | Per flush cycle |
| `[storage]` | `batch_flush_size_bytes` | Per flush cycle |
| `[storage]` | `max_open_fds` | Per LRU eviction check |
| `[deploy]` | `drain_timeout_secs` | On drain start |
| `[deploy]` | `shutdown_timeout_secs` | On shutdown start |
| `[deploy]` | `going_away_reason` | On drain start |
| `[[namespaces]]` | `max_total_messages` | Per publish |
| `[[namespaces]]` | `max_topics` | Per new topic |
| `[[namespaces]]` | `topic_max_depth` | Per publish |
| `[[namespaces]]` | `topic_max_bytes` | Per publish |
| `[[namespaces]]` | `backpressure_policy` | Per publish (at limit) |
| `[[namespaces]]` | `max_connections` | Per accept |
| `[[namespaces]]` | `max_disk_bytes` | Per publish |
| `[[namespaces]]` | `retention.*` | Per retention tick |
| `[[namespaces]]` | `max_redelivery_attempts` | Per redeliver |
| `[[namespaces]]` | `dlq_topic_suffix` | Per DLQ publish |

### Tier 2 — Reload with coordination (requires notification)

| Section | Field | Coordination needed |
|---------|-------|---------------------|
| `[[namespaces]]` | `api_key` | Auth store rebuild (`ArcSwap` on entries) |
| `[[namespaces]]` | `permissions` | Auth store rebuild |
| (new namespace) | (all fields) | Manager state creation |
| (removed namespace) | (all fields) | Rejected if active connections |

### Tier 3 — Restart only (ignored during hot-reload)

| Section | Field | Reason |
|---------|-------|--------|
| `[tcp]` | `bind_address` | Listener already bound |
| `[tcp]` | `port` | Listener already bound |
| `[http]` | `bind_address` | Listener already bound |
| `[http]` | `port` | Listener already bound |
| `[storage]` | `data_dir` | Paths baked into FsStorage |
| `[storage]` | `fsync_strategy` | Durability guarantee change |
| `[storage]` | `index_interval` | Existing segments use old interval |
| `[queue]` | `disk_backed` | Fundamental engine architecture |
| `[deploy]` | `admin_api_key` | Security-sensitive, restart-only by policy |
| `[hot_reload]` | `watch_file` | Watcher already started/not started |
| `[hot_reload]` | `debounce_secs` | Watcher already configured |
| `[metrics]` | (all fields) | Prometheus exporter initialized once at startup |
| `[logging]` | `format`, `file_enabled`, `file_dir`, `rotation`, `max_age_days`, `max_total_size_mb` | Tracing subscriber initialized once at startup |

> **Note on `logging.level`:** Already reloadable via `PUT /admin/log-level` (uses `tracing_subscriber::reload::Handle`). The config hot-reload should detect level changes in the TOML and forward them to the existing log level reload handle rather than duplicating the mechanism.

---

## CLAUDE.md Updates (after all phases)

After all phases are complete, add the following to CLAUDE.md:

- **Config Hot-Reload section**: Document `RuntimeConfig`, `SharedConfig`, `ArcSwap` pattern, subsystem reads
- **Admin Endpoints** (extend existing): `POST /admin/reload`, `GET /admin/config`
- **Observability Metrics** (extend existing): `fafb_config_reloads_total`, `fafb_config_reload_duration_seconds`, `fafb_config_last_reload_timestamp`, `fafb_config_last_reload_changes`
- **Constructor Patterns** (extend existing): `RuntimeConfig::from_server_config()`, `shared_config()`, `reload_config()`, `ApiKeyStore::spawn_reload_watcher()`, `DefaultNamespaceManager::register_namespace()`
- **Config Field Tiers**: Document the three-tier classification for future developers

---

## Phase 17.9 — Runtime Key Store Integration with Hot-Reload

### Goal

Ensure that config hot-reload correctly interacts with the runtime key store (`_system/keys.toml`) introduced in Plan 4 Phase 4.1. Config-file keys and runtime-created keys must coexist safely across reload cycles, and the `_system/keys.toml` path must be classified as Tier 3 (restart-only).

### Prerequisites

- Phase 17.7 complete (subsystem migration to `SharedConfig`)
- Plan 4 Phase 4.1 complete (`ApiKeyStore` with `ManagedKey`, `KeySource`, key store file)

### Design

#### Tier classification for key store

| Field | Tier | Reason |
|-------|------|--------|
| `{data_dir}/keys.toml` location | 3 (restart-only) | Derived from `data_dir`, which is Tier 3 |
| Runtime key content | N/A | Not in `config.toml` — managed via admin API, persisted to `_system/keys.toml` |

The key store file path (`{data_dir}/_system/keys.toml`) is **not a config field** — it's derived from `data_dir`. Since `data_dir` is Tier 3, the key store location cannot change at runtime.

#### Reload flow with runtime keys

When `reload_config()` fires (via HTTP or file watcher):

1. Rebuild config keys from the new `config.toml` (auto-named `_seed:{namespace}`)
2. Load current runtime keys from the `ArcSwap` store (filter `source: Runtime`) — NOT re-read from disk
3. Validate: no name collisions between new config keys and existing runtime keys
4. If a config reload **adds a namespace**: runtime keys for that namespace are unaffected (there likely are none yet)
5. If a config reload **removes a namespace**: runtime keys for that namespace become orphaned — log a warning per orphaned key, do NOT delete them. Operator must explicitly revoke via admin API.
6. If a config reload **changes a config key's permissions**: the `_seed:{namespace}` key entry is rebuilt with new permissions. Runtime keys are untouched.
7. Merge config keys + runtime keys, `ArcSwap::store()`

#### Name collision on reload

If a config reload introduces a namespace whose auto-generated config key name (`_seed:{namespace}`) collides with an existing runtime key name, the reload is **rejected** (same as other validation failures). This is extremely unlikely since `config-` prefix is reserved, but the validation must exist.

#### NamespaceAuthConfig changes

Currently, `NamespaceAuthConfig` stores a single `api_key` + `permissions`. With multi-key support, the `auth_changed` detection in `ConfigDiff` should only compare config-sourced keys — runtime keys are independent of config reload.

Update `RuntimeConfig::diff()` to:
- Compare `namespace_auth` (config keys only) as before
- Set `auth_changed = true` only when config-sourced auth data changes
- Runtime key mutations use their own `ArcSwap::store()` path, not the config reload path

### Tasks

- [ ] Update `ApiKeyStore::reload_from_config()` to preserve `source: Runtime` entries during config-triggered rebuilds
- [ ] Add orphan detection: on namespace removal, log warning for each runtime key whose namespace no longer exists in config
- [ ] Add name collision validation: reject reload if new config key name collides with existing runtime key name
- [ ] Update `RuntimeConfig::diff()` to only compare config-sourced auth data (ignore runtime keys for `auth_changed` flag)
- [ ] Add `_system/keys.toml` to file permissions warning on startup (reuse `check_config_permissions()`)
- [ ] Unit test: config reload preserves runtime keys unchanged
- [ ] Unit test: config reload with namespace removal logs warning for orphaned runtime keys
- [ ] Unit test: config reload with config key permission change updates only config keys
- [ ] Unit test: adding a namespace via reload does not affect runtime keys in other namespaces
- [ ] Unit test: name collision between new config key and existing runtime key rejects reload
- [ ] Integration test: create runtime key → reload config → runtime key still works for auth
- [ ] Integration test: create runtime key → remove namespace from config → reload → orphan warning logged, runtime key still in store

**Verification:** Config hot-reload and runtime key management coexist safely. Runtime keys survive config reloads. Orphaned keys are warned about, not silently deleted. All existing hot-reload tests pass unchanged.

## Out of Scope

- **YAML/JSON config formats** — TOML only. One format, zero ambiguity.
- **Config file generation wizard** — Operators copy the example file. No interactive setup.
- **Remote config sources** (etcd, Consul, S3) — Local file watching only. Remote sources are a separate concern.
- **Per-namespace config files** — All namespaces live in the same TOML file.
- **`server_id` field** — Useful for multi-instance, but not critical for single-server MVP.
- **Config versioning / rollback** — Forward-only. No rollback via admin endpoint.
- **Distributed config** (etcd, Consul) — Beyond local filesystem.
- **Gradual rollout** — All-or-nothing config application.
- **`max_message_size` reload for existing connections** — Would require reconstructing `MessageCodec` mid-connection. Low value, high complexity.
