# Phase 16: TLS / mTLS Support

## Goal

Enable encrypted and optionally mutually-authenticated connections on both the TCP and HTTP interfaces. Without TLS, no production deployment is possible — API keys transit in cleartext, message payloads are visible to any network observer, and compliance frameworks (SOC 2, HIPAA, PCI-DSS) categorically reject plaintext transport for sensitive data. mTLS adds a second layer: the server verifies the client's identity via certificate, enabling zero-trust service mesh deployments where API keys alone are insufficient.

## Guiding Principles

- **TLS is optional but loud.** Running without TLS must continue to work (development, CI, testing), but the server should log a startup warning when TLS is disabled. No silent insecurity.
- **One certificate, both listeners.** By default, TCP and HTTP share the same certificate and key. Operators who need separate certs (e.g., internal TCP, public HTTP) can override per-listener.
- **mTLS is additive.** TLS works without client certs. mTLS is a separate config knob (`client_ca_file`) — when set, the server requires and validates client certificates. When unset, standard TLS applies.
- **Hot reload of certificates.** Certificates expire and rotate. The server must support reloading cert/key files without a restart (via SIGHUP or admin endpoint), or at minimum detect expiry and log errors.
- **No custom crypto.** Use `rustls` (not OpenSSL bindings). Pure Rust, audited, no C dependency chain. `tokio-rustls` for async integration. `rcgen` only for test cert generation.
- **Protocol unchanged.** TLS wraps the transport — the framing protocol above it is byte-identical. Clients negotiate TLS before sending `Connect`. No STARTTLS (upgrade-in-place) — the port is either TLS or plaintext.

## Current State

| Capability | Implementation | Gap |
|---|---|---|
| TCP transport | Raw `TcpStream` via tokio | No encryption |
| HTTP transport | `axum 0.8` on `hyper` with plain HTTP via `axum::serve()` | No HTTPS — does not use `axum-server` |
| Auth | API key in `Connect` frame / `Authorization` header | Key transmitted in cleartext |
| Client identity | API key only | No certificate-based identity |
| Connection handler | `run_connection(stream: TcpStream, ...)` — concrete type | Must become generic over `AsyncRead + AsyncWrite` |
| HTTP graceful shutdown | `axum::serve().with_graceful_shutdown()` via `ServerLifecycle` watch channel | `axum-server` has different shutdown API — must preserve lifecycle integration |
| HTTP test helper | `run_with_listener(listener: TcpListener)` for port-0 binding | Must keep working for non-TLS test paths |

## Design

### Configuration

```toml
[tls]
# Shared TLS config — applies to both TCP and HTTP unless overridden
enabled = false                    # Master switch
cert_file = "/etc/fafb-q/server.crt"   # PEM certificate chain
key_file = "/etc/fafb-q/server.key"    # PEM private key
client_ca_file = ""                # PEM CA bundle for mTLS (empty = no client auth)
min_version = "1.2"                # Minimum TLS version: "1.2" or "1.3"
reload_interval_secs = 3600        # Check cert files for changes (0 = no auto-reload)
ciphersuite_allowlist = []         # Empty = rustls defaults. When set, only these ciphersuites are offered.

[tcp]
# tls_cert_file / tls_key_file — override shared cert for TCP listener only
# tls_client_ca_file — override shared client CA for TCP listener only

[http]
# tls_cert_file / tls_key_file — override shared cert for HTTP listener only
# tls_client_ca_file — override shared client CA for HTTP listener only
```

**Config validation (startup):**
- `enabled = true` requires `cert_file` and `key_file` to be non-empty and readable
- `min_version` must be `"1.2"` or `"1.3"` — reject anything else
- `client_ca_file` when non-empty must be readable and contain at least one valid PEM certificate
- Per-listener overrides (`tls_cert_file`, etc.) are validated the same way when present
- Validation error = fatal startup failure with specific error message (not a generic "TLS config invalid")

**Environment variable overrides:**
- `FAFB_TLS_CERT_FILE`, `FAFB_TLS_KEY_FILE`, `FAFB_TLS_CLIENT_CA_FILE`
- Env vars override `[tls]` section values but NOT per-listener overrides

### TLS Module (`src/tls.rs`)

```rust
use std::sync::Arc;
use arc_swap::ArcSwap;
use tokio_rustls::TlsAcceptor;
use rustls::ServerConfig;

pub struct TlsProvider {
    acceptor: ArcSwap<TlsAcceptor>,  // hot-swappable via arc_swap
    server_config: ArcSwap<Arc<ServerConfig>>,  // exposed for HTTP/axum-server
    client_auth: bool,
    config: TlsConfig,  // retained for reload — knows file paths
    cert_mtime: AtomicU64,  // last observed mtime (unix secs) for change detection
    key_mtime: AtomicU64,
}

impl TlsProvider {
    /// Build from config. Returns None if TLS is disabled.
    /// Validates cert/key readability and match at construction time.
    pub fn from_config(config: &TlsConfig) -> Result<Option<Self>, Error>;

    /// Reload certificate and key files from disk.
    /// Called by SIGHUP handler or background timer.
    /// Returns Ok(true) if certs were reloaded, Ok(false) if unchanged.
    pub fn reload(&self) -> Result<bool, Error>;

    /// Wrap a raw TcpStream in TLS. Returns the negotiated stream.
    pub async fn accept(&self, stream: TcpStream) -> Result<TlsStream<TcpStream>, Error>;

    /// Return a clone of the current ServerConfig for axum-server.
    pub fn server_config(&self) -> Arc<ServerConfig>;

    /// Extract client certificate info from a completed TLS connection.
    /// Returns None if mTLS is not active or client presented no cert.
    pub fn client_info(conn: &TlsStream<TcpStream>) -> Option<ClientCertInfo>;

    /// Seconds until the current server cert expires. 0 if already expired.
    pub fn cert_ttl_secs(&self) -> u64;
}

/// Extracted client certificate metadata for logging and audit.
pub struct ClientCertInfo {
    pub common_name: Option<String>,
    pub sans: Vec<String>,           // Subject Alternative Names
    pub fingerprint_sha256: String,  // hex-encoded SHA-256 of DER cert
    pub serial: String,              // hex-encoded serial number
    pub not_after: u64,              // expiry as unix millis
}
```

### TCP Integration — Stream Abstraction

**Key change:** `run_connection()` currently takes `stream: TcpStream` (concrete type). `Framed<S, MessageCodec>` already supports any `S: AsyncRead + AsyncWrite + Unpin`, so the fix is making `run_connection` generic:

```rust
// src/server/tcp/connection.rs

pub(crate) async fn run_connection<S, M>(
    stream: S,
    config: ConnectionConfig,
    manager: Arc<M>,
)
where
    S: tokio::io::AsyncRead + tokio::io::AsyncWrite + Unpin + Send + 'static,
    M: NamespaceManager + 'static,
{
    let mut framed = Framed::new(stream, MessageCodec::new(max_frame_size));
    // ... rest unchanged
}
```

**Why generic over trait object:** Avoids `Box<dyn AsyncRead + AsyncWrite>` heap allocation and vtable dispatch on every read/write in the hot path. `TlsStream<TcpStream>` and `TcpStream` are both concrete at compile time — the compiler monomorphizes `run_connection` for each, producing optimal code for both paths.

```rust
// src/server/tcp.rs — accept loop changes

let tls_provider = TlsProvider::from_config(&config.tls)?;

if config.tls.enabled && tls_provider.is_none() {
    // Should not happen due to validation, but belt-and-suspenders
    return Err(anyhow!("TLS enabled but provider failed to initialize"));
}

if tls_provider.is_none() {
    tracing::warn!("TLS is disabled — connections are unencrypted. \
                    Set [tls] enabled=true with cert_file/key_file for production use.");
}

loop {
    let (raw_stream, peer_addr) = listener.accept().await?;

    match &tls_provider {
        Some(tls) => {
            let tls = Arc::clone(tls);
            tokio::spawn(async move {
                match tls.accept(raw_stream).await {
                    Ok(tls_stream) => {
                        let cert_info = TlsProvider::client_info(&tls_stream);
                        // Add cert CN to connection span if available
                        run_connection(tls_stream, conn_config, manager).await;
                    }
                    Err(e) => {
                        tracing::debug!(peer = %peer_addr, error = %e, "TLS handshake failed");
                        // metric: fafb_tls_handshake_total{status="error"}
                    }
                }
            });
        }
        None => {
            tokio::spawn(async move {
                run_connection(raw_stream, conn_config, manager).await;
            });
        }
    }
}
```

**TLS handshake timeout:** The `tls.accept()` call should have a timeout (e.g., 10 seconds) to prevent slow-handshake DoS. Wrap with `tokio::time::timeout()`. Configurable via `TlsConfig.handshake_timeout_secs` (default 10).

### HTTP Integration

**Important:** The current HTTP server uses `axum::serve()` with `with_graceful_shutdown()` tied to `ServerLifecycle`'s watch channel. Switching to `axum-server` changes the shutdown API. Two options:

**Option A — `axum-server` (preferred for TLS):**
```rust
// src/server/http.rs — TLS-aware serve

async fn serve(&self, listener: TcpListener) -> Result<()> {
    let router = self.router();

    match &self.tls_provider {
        Some(tls) => {
            let rustls_config = RustlsConfig::from_config(tls.server_config());
            let addr = listener.local_addr()?;
            // axum-server has its own graceful shutdown via handle
            let handle = axum_server::Handle::new();
            let shutdown_handle = handle.clone();
            let mut lifecycle_rx = self.state.lifecycle.subscribe();

            tokio::spawn(async move {
                loop {
                    if lifecycle_rx.changed().await.is_err() { break; }
                    if *lifecycle_rx.borrow() >= ServerState::ShuttingDown { break; }
                }
                shutdown_handle.graceful_shutdown(Some(Duration::from_secs(10)));
            });

            axum_server::bind_rustls(addr, rustls_config)
                .handle(handle)
                .serve(router.into_make_service())
                .await?;
        }
        None => {
            // Existing path unchanged — use axum::serve()
            let mut lifecycle_rx = self.state.lifecycle.subscribe();
            axum::serve(listener, router)
                .with_graceful_shutdown(async move {
                    loop {
                        if lifecycle_rx.changed().await.is_err() { break; }
                        if *lifecycle_rx.borrow() >= ServerState::ShuttingDown { break; }
                    }
                })
                .await?;
        }
    }
    Ok(())
}
```

**Option B — `hyper-rustls` + manual `TlsAcceptor`:**
Keep `axum::serve()` for both paths by accepting TLS at the listener level before passing to axum. This avoids the `axum-server` dependency but requires more manual plumbing. Less recommended because `axum-server` handles edge cases (e.g., TLS handshake backpressure).

**Decision: Use Option A.** The non-TLS path stays on `axum::serve()` (zero behavioral change for existing users), and the TLS path uses `axum-server` which is purpose-built for this.

**`run_with_listener` for tests:** Keep the existing non-TLS test path. Add `run_with_tls_listener` for TLS integration tests that pre-bind and pass a `TlsAcceptor`.

### mTLS Flow

1. Server loads `client_ca_file` into `rustls::ServerConfig` with `WebPkiClientVerifier`
2. `WebPkiClientVerifier::builder(root_store).build()` — requires `rustls 0.23` API
3. During TLS handshake, server requests client certificate
4. Client presents cert signed by the trusted CA
5. After handshake, `TlsProvider::client_info()` extracts CN, SANs, fingerprint from `peer_certificates()`
6. Cert info is logged in the connection span: `info_span!("conn", client, peer, cert_cn)`
7. API key auth still required — mTLS authenticates the transport, API keys authorize the namespace

**mTLS + API key interaction:** Both are required when mTLS is enabled. mTLS validates "this client has a trusted certificate" (transport-level identity). API key validates "this client is authorized for this namespace" (application-level authorization). They serve different purposes and neither replaces the other.

**CRL/OCSP:** Out of scope for initial implementation. Note: `rustls 0.23` supports CRL checking via `WebPkiClientVerifier::builder().with_crls()`. Can be added as a follow-up without protocol changes.

### Certificate Reload

Two mechanisms, both calling `TlsProvider::reload()`:

1. **SIGHUP handler** — operator sends `kill -HUP <pid>`, server reloads certs immediately
2. **Background timer** — every `reload_interval_secs`, stat the cert/key files, reload if mtime changed

Reload is atomic via `arc_swap::ArcSwap<TlsAcceptor>` — existing connections keep their TLS session, new connections use the new cert.

**Reload for HTTP:** HTTP uses `TlsListener` which loads the current acceptor from `TlsProvider` via `ArcSwap` on each `accept()` call. `TlsProvider::reload()` swaps the acceptor atomically — new HTTPS connections automatically use the new cert with no additional wiring. When TCP and HTTP share a `TlsProvider` (the common case), a single `reload()` updates both.

**Admin endpoint:** `POST /admin/tls/reload` — trigger immediate cert reload via HTTP (requires admin API key). Returns `200 { "status": "reloaded", "cert_expires": "ISO8601" }` or `500 { "error": "..." }`. This supplements SIGHUP for environments where signal delivery is awkward (e.g., container orchestrators).

**Crash isolation for reload task:** The background cert reload timer should be monitored in the same `select!` loop as other background tasks (retention, compaction). If the reload task panics, respawn with exponential backoff per the existing crash isolation pattern.

### Connection Span Enhancement

Current connection span: `info_span!("conn", client = %client_id, peer = %peer_addr)`

With TLS, extend to include TLS metadata:
```rust
info_span!(
    "conn",
    client = %client_id,
    peer = %peer_addr,
    tls = tls_active,                          // bool
    tls.version = tls_version_str,             // "TLSv1.3" or ""
    tls.cipher = cipher_suite_str,             // "TLS13_AES_256_GCM_SHA384" or ""
    cert.cn = cert_info.as_ref().map(|c| c.common_name.as_deref().unwrap_or("")),
)
```

This enables log filtering by TLS status and cert identity across all log lines for a connection.

### Metrics

- `fafb_tls_handshake_total{status}` — success/failure counter
- `fafb_tls_handshake_duration_seconds` — histogram
- `fafb_tls_cert_expiry_seconds` — gauge, seconds until server cert expires (for alerting). Updated on load/reload.
- `fafb_tls_reload_total{status}` — success/failure counter for cert reloads
- `fafb_mtls_auth_total{status}` — mTLS client auth success/failure (status: success, no_cert, untrusted, expired)
- `fafb_tls_version_total{version}` — counter per TLS version negotiated (1.2 vs 1.3) — useful for deprecation rollout
- `fafb_tls_handshake_timeout_total` — counter for handshakes that exceeded the timeout

### Error Handling

| Scenario | Behavior |
|---|---|
| TLS enabled but cert file missing | Fatal startup error |
| TLS enabled but key file doesn't match cert | Fatal startup error |
| TLS enabled but cert is expired at startup | Fatal startup error (warn + refuse to start with expired cert) |
| TLS enabled but cert expires within 7 days | Startup warning: "certificate expires in N days" |
| Client connects with plaintext to TLS port | TLS handshake fails, connection closed, metric incremented |
| TLS handshake exceeds timeout | Connection closed, `fafb_tls_handshake_timeout_total` incremented |
| mTLS enabled, client has no cert | Handshake rejected, `fafb_mtls_auth_total{status="no_cert"}` |
| mTLS enabled, client cert not signed by trusted CA | Handshake rejected, `fafb_mtls_auth_total{status="untrusted"}` |
| mTLS enabled, client cert expired | Handshake rejected, `fafb_mtls_auth_total{status="expired"}` |
| Cert reload fails (bad file) | Warning logged, old cert continues to serve, `fafb_tls_reload_total{status="error"}` |

## Implementation Phases

### Phase 16.1 — Stream Genericization & TLS Foundation

**Goal:** Make `run_connection` generic over the stream type and add TLS config/module scaffolding. All existing tests must pass after this phase — no TLS connections yet.

- [x] Add dependencies to `Cargo.toml`: `rustls = "0.23"`, `tokio-rustls = "0.26"`, `arc-swap = "1"`, `rustls-pemfile = "2"`
- [x] Add dev dependency: `rcgen = { version = "0.13", features = ["pem"] }` for test cert generation
- [x] Add `TlsConfig` struct to `src/config.rs` with the `[tls]` TOML section (see Design > Configuration above)
  - [x] Fields: `enabled`, `cert_file`, `key_file`, `client_ca_file`, `min_version`, `reload_interval_secs`, `ciphersuite_allowlist`, `handshake_timeout_secs`
  - [x] Per-listener override fields on `TcpConfig` and `HttpConfig`: `tls_cert_file`, `tls_key_file`, `tls_client_ca_file`
  - [x] Environment variable overrides: `FAFB_TLS_CERT_FILE`, `FAFB_TLS_KEY_FILE`, `FAFB_TLS_CLIENT_CA_FILE`
  - [x] Config validation: `enabled = true` requires readable `cert_file`/`key_file`, `min_version` must be `"1.2"` or `"1.3"`, `client_ca_file` when set must contain valid PEM — fatal startup error with specific message on failure
- [x] Genericize `run_connection` in `src/server/tcp/connection.rs`: change `stream: TcpStream` to `stream: S` where `S: AsyncRead + AsyncWrite + Unpin + Send + 'static`
  - [x] Verify `Framed<S, MessageCodec>` compiles with the new generic bound
  - [x] Run full test suite — all existing tests must pass with zero changes
- [x] Create `src/tls.rs` with `TlsProvider` skeleton
  - [x] `TlsProvider` struct with `ArcSwap<TlsAcceptor>`, config, mtime tracking fields
  - [x] `from_config(config: &TlsConfig) -> Result<Option<Self>>`: return `None` when `enabled = false`, build `ServerConfig` with cert/key when enabled
  - [x] `accept(stream: TcpStream) -> Result<TlsStream<TcpStream>>`: wrap raw stream with TLS acceptor
  - [x] `server_config() -> Arc<ServerConfig>`: return current config for HTTP/axum-server use
  - [x] `cert_ttl_secs() -> u64`: parse cert expiry from loaded certificate
  - [x] Startup warning when cert expires within 7 days, fatal error when already expired
- [x] Unit tests for `TlsProvider`
  - [x] Construction from valid config with `rcgen`-generated cert+key
  - [x] Construction fails with missing cert file
  - [x] Construction fails with key/cert mismatch
  - [x] Construction fails with expired cert
  - [x] `cert_ttl_secs()` returns correct value
  - [x] Config validation: invalid `min_version`, unreadable files, empty `client_ca_file` path

**Verification:** `cargo test` passes. `cargo clippy` clean. No TLS connections are made yet — this phase is pure scaffolding.

**Post-phase improvements (applied):**
- Replaced hand-rolled ASN.1/DER parser with `x509-parser` crate for robust X.509 parsing.
- Cached cert expiry as `AtomicU64` in `TlsProvider` — `cert_ttl_secs()` now reads from cache (zero disk I/O).
- Added `Error::Tls(String)` variant for runtime TLS errors (handshake failures), separating them from `Error::Config`.
- Added `TlsConfig::resolved_for_listener()` — merges shared `[tls]` config with per-listener overrides.
- Added `reload()` tests: detects changed cert files, fails gracefully with corrupt cert, verifies cached expiry survives file deletion.

---

### Phase 16.2 — TLS on TCP

**Implementation notes from Phase 16.1:**
- `run_connection` is already generic over `S: AsyncRead + AsyncWrite + Unpin + Send + 'static` — no further changes needed there.
- Use `TlsConfig::resolved_for_listener(tcp.tls_cert_file.as_ref(), tcp.tls_key_file.as_ref(), tcp.tls_client_ca_file.as_ref())` to build the per-listener config before passing to `TlsProvider::from_config()`.
- `TlsProvider::accept()` returns `Error::Tls`, not `Error::Config` — match on this for the handshake failure metric path.
- The connection-limit rejection path in `tcp.rs` (lines 153-161) wraps raw `TcpStream` in `Framed` to send an error before closing. This stays as-is because TLS handshake hasn't happened yet at that point — we reject before the TLS overhead.
- `main.rs` should construct the `TlsProvider` and pass `Option<Arc<TlsProvider>>` to both `TcpServer` and `HttpServer`. If the TCP and HTTP listeners use different certs (per-listener overrides), construct two separate providers.

**Goal:** TCP accept loop wraps connections with TLS when enabled. Clients can connect, authenticate, and publish/subscribe over encrypted connections.

- [x] Modify TCP accept loop in `src/server/tcp.rs` to use `TlsProvider`
  - [x] If TLS enabled: call `tls.accept(raw_stream)` wrapped in `tokio::time::timeout(handshake_timeout)`
  - [x] If TLS disabled: pass raw stream directly to `run_connection` (existing path, unchanged)
  - [x] Log `tracing::warn!` at startup when TLS is disabled ("connections are unencrypted")
  - [x] Log `tracing::debug!` on TLS handshake failure with peer address and error
- [x] Thread `Option<Arc<TlsProvider>>` through `TcpServer` construction
  - [x] `TcpServer::new()` accepts optional `TlsProvider`
  - [x] `ConnectionConfig` does not need `TlsProvider` — TLS wrapping happens before `run_connection`
- [x] Add TLS metrics
  - [x] `fafb_tls_handshake_total{status}` counter (success / error / timeout)
  - [x] `fafb_tls_handshake_duration_seconds` histogram
  - [x] `fafb_tls_handshake_timeout_total` counter
  - [ ] `fafb_tls_version_total{version}` counter per negotiated TLS version (1.2 vs 1.3) — version is now extracted in 16.4 for span fields (`tls.version`); adding the metric counter is trivial but deferred
- [x] Integration test: TLS connect + auth + publish + subscribe using `rcgen` test certs
  - [x] Create shared test helper: `generate_test_certs()` in `tests/common/`
  - [x] Build `TlsConnector` in test client for TLS connections (`make_test_tls_connector()`)
  - [x] Full round-trip: connect over TLS → Connect frame → publish → subscribe → receive message
- [x] Integration test: plaintext client to TLS port — handshake fails, clean disconnect
- [x] Integration test: TLS handshake timeout — slow client, verify timeout fires

**Post-phase notes (applied):**
- Added `TlsProvider::handshake_timeout()` method to expose configured timeout duration.
- TLS handshake wrapping happens in the TCP accept loop (spawned task), not in `run_connection` — connection handler remains TLS-agnostic.
- `TestClient<S>` is generic over stream type with default `S = TcpStream`. `TlsTestClient` is a type alias for `TestClient<TlsStream<TcpStream>>`. Shared `send()`/`recv()`/`recv_timeout()` methods via generic impl; constructors are type-specific (`connect()` vs `connect_tls()`).
- `start_test_tls_tcp_server_with_config(TlsConfig)` helper for tests that need custom TLS settings (e.g., short handshake timeout).
- `make_test_tls_connector()` explicitly uses `ring` crypto provider via `builder_with_provider()` to avoid process-global `CryptoProvider` issues in parallel tests.
- Connection count (`inc_connections`/`dec_connections`) only tracks successfully established connections — TLS handshake failures and timeouts do not inflate the active connection count.
- TLS-disabled warning only logs in `run()` (production path), not `run_with_listener()` (test path).
- TLS metrics: `fafb_tls_handshake_total{status=success|error|timeout}`, `fafb_tls_handshake_duration_seconds`, `fafb_tls_handshake_timeout_total`.
- `main.rs` constructs `TlsProvider` from `resolved_for_listener()` config and passes to `TcpServer`; fatal exit on config error.

**Verification:** `cargo test` passes for both TLS and non-TLS paths. Existing non-TLS integration tests still pass without modification.

---

### Phase 16.3 — TLS on HTTP (HTTPS)

**Goal:** HTTP server serves over HTTPS when TLS is enabled. Plaintext path stays on `axum::serve()` with zero behavioral change.

- [x] Wire `Option<Arc<TlsProvider>>` into `HttpServer`
  - [x] Add `tls_provider` field to `HttpServer` struct
  - [x] `HttpServer::new()` accepts optional provider as third argument
- [x] Implement branching `serve()` method in `src/server/http.rs`
  - [x] TLS path: custom `TlsListener` implementing `axum::serve::Listener` wraps `TcpListener` + `TlsProvider` with handshake timeout
  - [x] TLS path: same `axum::serve()` + `with_graceful_shutdown()` — identical shutdown code for both paths
  - [x] Plaintext path: existing `axum::serve()` + `with_graceful_shutdown()` — completely unchanged
- [x] Ensure `run_with_listener` continues to work for both TLS and non-TLS paths (TLS-aware based on `tls_provider`)
- [x] `main.rs` constructs HTTP `TlsProvider` from `resolved_for_listener()` config before HTTP server start
- [x] Integration test: HTTPS publish endpoint — publish message over HTTPS, verify stored
- [x] Integration test: HTTPS fetch endpoint — publish then fetch messages over HTTPS
- [x] Integration test: HTTPS health endpoints (`/health/live`, `/health/ready`, `/health`)
- [x] Integration test: HTTPS admin endpoints (`/admin/drain/status` with and without admin key)
- [x] Integration test: graceful shutdown with HTTPS — verify server stops accepting after shutdown
- [x] Integration test: plaintext client to HTTPS port — handshake fails, server stays healthy
- [x] Integration test: HTTPS handshake timeout — raw TCP socket never completes handshake, concurrent connections unblocked
- [x] `TlsListener` performs TLS handshakes concurrently via `JoinSet` (prevents slow-handshake DoS)
- [x] `TlsConfig` derives `PartialEq`; `main.rs` shares single `Arc<TlsProvider>` when TCP and HTTP resolved configs are identical

**Post-phase notes (applied):**
- Used custom `TlsListener` implementing `axum::serve::Listener` instead of `axum-server` dependency. This avoids pulling in `aws-lc-rs` via Cargo feature unification (which would conflict with the project's `ring`-only crypto policy and require a C compiler). Zero new dependencies.
- `TlsListener` performs TLS handshakes concurrently via `JoinSet`. `accept()` uses biased `select!` to drain completed handshakes before accepting new TCP connections. A slow/malicious client performing a partial handshake does not block other connections.
- Same `axum::serve()` + `with_graceful_shutdown()` code for both TLS and plaintext paths. No separate shutdown API needed.
- `run_with_listener()` is TLS-aware: when `tls_provider` is `Some`, it wraps the listener in `TlsListener`. Both existing plaintext test paths and new HTTPS tests use `run_with_listener()`.
- TLS handshake metrics (`fafb_tls_handshake_total{status}`, `fafb_tls_handshake_duration_seconds`, `fafb_tls_handshake_timeout_total`) shared with TCP listener — same metric names, cumulative across both listeners.
- Added `127.0.0.1` IP SAN to `generate_test_certs()` so HTTPS tests can use `https://127.0.0.1:{port}` without DNS resolution issues.
- `reqwest` test client trusts the test root cert via `add_root_certificate(Certificate::from_der(...))`. Uses the existing native-tls backend (no new TLS dependency for test client).
- `TlsConfig` derives `PartialEq`. `main.rs` compares resolved TCP and HTTP TLS configs — when identical (no per-listener overrides), a single `Arc<TlsProvider>` is shared. Simplifies Phase 16.5 cert reload.

**Verification:** All HTTP tests pass in both TLS and plaintext modes. Deploy integration tests still pass. 11 TLS tests total (4 TCP + 7 HTTPS).

---

### Phase 16.4 — mTLS (Mutual TLS)

**Goal:** When `client_ca_file` is configured, the server requires and validates client certificates on both TCP and HTTP. API key auth remains required alongside mTLS.

- [x] Add `WebPkiClientVerifier` support to `TlsProvider`
  - [x] When `client_ca_file` is set: load CA bundle into `RootCertStore`, build `WebPkiClientVerifier::builder_with_provider(root_store, ring_provider).build()`
  - [x] Attach client verifier to `ServerConfig`
  - [x] `client_auth: bool` field on `TlsProvider` to indicate mTLS is active
- [x] Implement `TlsProvider::client_info(conn: &TlsStream<TcpStream>) -> Option<ClientCertInfo>`
  - [x] Extract `peer_certificates()` from completed TLS connection
  - [x] Parse common name (CN) from subject
  - [x] Extract Subject Alternative Names (SANs) — DNS names and IPv4 addresses
  - [x] Compute SHA-256 fingerprint of DER cert (hex-encoded, colon-separated) using `ring::digest`
  - [x] Extract serial number (hex-encoded, colon-separated)
  - [x] Extract `not_after` as unix millis
- [x] Add `ClientCertInfo` struct (see Design section)
- [x] Thread client cert info through TCP connection setup
  - [x] After successful TLS handshake in accept loop, call `TlsProvider::client_info()`
  - [x] Log cert CN, serial, and fingerprint at debug level when mTLS client cert is present
- [x] Extend connection `info_span!` with TLS metadata
  - [x] Add fields: `tls` (bool), `cert.cn` — deferred values recorded after TLS handshake
  - [x] Plaintext connections record `tls = false`; TLS connections record `tls = true`
  - [x] `tls.version` — recorded from `server_conn.protocol_version()` (TLSv1.2 / TLSv1.3)
  - [x] `tls.cipher` — recorded from `server_conn.negotiated_cipher_suite().suite()` (Debug format)
- [x] Integration tests for TCP mTLS
  - [x] Valid client cert signed by trusted CA → accepted, full publish round-trip works
  - [x] Client presents no cert → connection rejected (server-side handshake failure surfaces on first I/O)
  - [x] Client cert signed by untrusted CA → connection rejected
  - [ ] Client cert expired → deferred (requires rcgen expired-cert generation with CA signing)
- [x] Integration tests for HTTP mTLS
  - [x] HTTPS request with valid client cert → publish succeeds
  - [x] HTTPS request without client cert → handshake rejected
- [ ] `fafb_mtls_auth_total{status}` metric — deferred. mTLS failures already surface as `fafb_tls_handshake_total{status="error"}` since rustls rejects during TLS handshake. Fine-grained breakdown (no_cert/untrusted/expired) requires parsing error strings which is fragile.

**Post-phase notes (applied):**
- `WebPkiClientVerifier::builder_with_provider()` used instead of `builder()` to avoid process-level `CryptoProvider` ambiguity when `aws-lc-rs` is also linked (via `reqwest`'s `rustls-tls` feature in dev-deps).
- `client_info()` uses `ring::digest::SHA256` for fingerprint computation — `ring` added as a direct dependency (was already transitively linked via `rustls`).
- Connection `info_span!` uses `tracing::field::Empty` for deferred recording: `tls` and `cert.cn` are recorded inside the spawned task after the TLS handshake completes. This keeps TLS metadata in the span for the entire connection lifetime.
- mTLS rejection tests verify that connections fail at the I/O level, not necessarily at the `connect()` call. In TLS, the server's certificate rejection alert may be buffered — the client-side `connect()` can appear to succeed, but the first `send()`/`recv()` will fail. Tests handle both immediate handshake failure and deferred I/O failure.
- `reqwest` dev-dependency changed to `default-features = false, features = ["json", "rustls-tls-manual-roots"]` to support `Identity::from_pem()` for mTLS HTTPS test clients. Uses `rustls-tls-manual-roots` instead of `rustls-tls` to avoid pulling in `webpki-roots` (tests use custom CAs). `aws-lc-rs` still appears transitively via `hyper-rustls` Cargo feature unification — unavoidable without changing upstream deps.
- `ensure_ring_provider()` helper in test infrastructure installs the `ring` process-level default to avoid `CryptoProvider` auto-detection failures in parallel tests.
- Test cert generation: `generate_mtls_test_certs()` creates a self-signed CA, server cert signed by CA (SAN=localhost+127.0.0.1), valid client cert (CN=test-client), and untrusted client cert (signed by a different rogue CA). All certs generated via `rcgen` at test time.
- Pre-existing `HttpServer::new` signature mismatch in `lifecycle_integration.rs` fixed (added missing `None` TLS provider argument).
- `client_info()` refactored: cert-parsing logic extracted to `parse_client_cert_info(der: &[u8])` for unit testability. 8 unit tests cover CN extraction, DNS/IP SANs, SHA-256 fingerprint format, serial format, expiry, missing CN, and garbage input.
- `tls.version` and `tls.cipher` span fields added to TCP connection span. Extracted from `ServerConnection::protocol_version()` and `negotiated_cipher_suite().suite()` after handshake. `tls.version` recorded as human-readable string ("TLSv1.2" / "TLSv1.3"). `tls.cipher` recorded as `Debug` format of `CipherSuite`.

**Verification:** mTLS works on both TCP and HTTP. API key auth still required alongside mTLS. All 16 TLS tests pass (11 existing + 5 new mTLS). 8 new `parse_client_cert_info` unit tests pass. All non-mTLS tests still pass without modification. `cargo clippy` clean.

---

### Phase 16.5 — Certificate Hot Reload

**Goal:** Certificates can be reloaded without server restart via SIGHUP, background timer, or admin endpoint. Existing connections are unaffected — only new connections use the new cert.

- [x] Implement `TlsProvider::reload() -> Result<bool>`
  - [x] Stat cert/key files, compare mtime against `cert_mtime`/`key_mtime` `AtomicU64` fields
  - [x] If unchanged, return `Ok(false)` (no-op)
  - [x] If changed, reload cert+key from disk, build new `ServerConfig`, swap via `ArcSwap`
  - [x] If reload fails (bad file, parse error), log warning and return `Err` — old cert continues serving
  - [x] Update `cert_mtime`/`key_mtime` on successful reload
- [x] SIGHUP handler integration
  - [x] Extend existing signal handling in `main.rs` to catch SIGHUP
  - [x] On SIGHUP, call `tls_provider.reload()` and log result
- [x] Background timer task for automatic reload
  - [x] Spawn task that checks every `reload_interval_secs` (skip if 0)
  - [x] Crash-isolate with exponential backoff respawn (same pattern as retention/compaction tasks)
  - [x] Monitor `JoinHandle` in the main `select!` loop
- [x] Certificate expiry gauge
  - [x] `fafb_tls_cert_expiry_seconds` gauge — seconds until server cert expires
  - [x] Update on initial load and every successful reload
- [x] `fafb_tls_reload_total{status}` metric (success / error / unchanged)
- [x] Admin endpoint: `POST /admin/tls/reload`
  - [x] Requires admin API key (same auth as existing admin endpoints)
  - [x] Triggers immediate `tls_provider.reload()`
  - [x] Returns `200 { "providers": [{ "listener", "status", "cert_expires" }] }` — per-provider results
  - [x] Returns `500` when any provider fails (with per-provider error details)
  - [x] Returns `404` when TLS is not enabled
- [x] Admin endpoint: `GET /admin/tls/status`
  - [x] Returns cert details (listener, cert_file, cert_expires, cert_ttl_secs, fingerprint, client_auth, min_version) without triggering reload
  - [x] Returns `404` when TLS is not enabled
- [x] Verify reload propagates to both TCP and HTTP listeners
  - [x] When shared `TlsProvider`: single `reload()` call updates both listeners (via `ArcSwap`)
  - [x] When separate providers (per-listener overrides): reload each independently
  - [x] HTTP `TlsListener` loads the current acceptor from `TlsProvider` on each `accept()` — no additional wiring needed
- [x] Integration test: swap cert file on disk → trigger reload → verify new cert detected with different expiry
- [x] Integration test: reload with invalid/corrupt cert file → verify old cert continues serving, server still responds
- [x] Integration test: `POST /admin/tls/reload` triggers reload and returns per-provider results
- [x] Integration test: `POST /admin/tls/reload` requires admin auth (403 without)
- [x] Integration test: `GET /admin/tls/status` returns cert details (fingerprint, TTL, expiry, etc.)
- [x] Integration test: `GET /admin/tls/status` requires admin auth (403 without)
- [ ] Integration test: SIGHUP triggers reload — deferred (requires spawning a real process and sending signal; complex to test in-process)

**Post-phase notes (applied):**
- `TlsProvider::reload()` was already implemented in Phase 16.1 — this phase wired it into all three trigger mechanisms (SIGHUP, timer, admin endpoint).
- `reload_tls_providers()` helper in `main.rs` — shared by SIGHUP handler and background timer. Logs results and updates metrics per-provider.
- `LabeledTlsProvider` pairs a `TlsProvider` with a listener label (`"shared"`, `"tcp"`, `"http"`). `TlsProviders(Vec<LabeledTlsProvider>)` injected via `axum::Extension` on admin routes — not stored in `AppState` (avoids cloning on every request and touching every test construction site).
- `POST /admin/tls/reload` returns per-provider results: `{ "providers": [{ "listener": "shared", "status": "success", "cert_expires": "..." }] }`.
- `GET /admin/tls/status` returns cert details per provider: listener, cert_file, cert_expires, cert_ttl_secs, cert_fingerprint_sha256, client_auth, min_version. No reload triggered — safe for dashboards.
- `TlsProvider::cert_fingerprint()` returns cached SHA-256 fingerprint (colon-separated hex), updated on construction and reload. Uses `RwLock<String>` since reads are rare (admin API only).
- Background cert reload task follows the existing crash isolation pattern: `spawn_cert_reload` closure, `JoinHandle` monitored in main `select!` loop, exponential backoff respawn on panic.
- `fafb_tls_cert_expiry_seconds{listener}` gauge labeled per listener (shared/tcp/http). Set on startup and after each successful reload. `fafb_tls_reload_total{status,listener}` counter labeled per listener.
- SIGHUP handler does NOT trigger shutdown — only reloads certs. SIGINT/SIGTERM still trigger graceful shutdown.

**Verification:** All 22 TLS tests pass (16 existing + 6 new reload/status tests). All non-TLS integration tests pass (576 total). `cargo clippy` clean (no new warnings).

## Dependencies

```toml
# Main dependencies
rustls = { version = "0.23", default-features = false, features = ["ring", "logging", "std", "tls12"] }
tokio-rustls = { version = "0.26", default-features = false, features = ["ring"] }
arc-swap = "1"
rustls-pemfile = "2"
x509-parser = "0.16"

# Dev dependencies (test only)
rcgen = { version = "0.13", features = ["pem"] }
```

**Compatibility notes:**
- Uses `ring` crypto backend exclusively (no `aws-lc-rs`, no C compiler needed). Both `rustls` and `tokio-rustls` are configured with `default-features = false` + `ring`.
- HTTPS uses a custom `TlsListener` implementing `axum::serve::Listener` (axum 0.8) instead of `axum-server`. This avoids pulling in `aws-lc-rs` via Cargo feature unification.
- `main.rs` shares a single `Arc<TlsProvider>` between TCP and HTTP when their resolved configs are identical (the common case — no per-listener overrides).

## Testing Strategy

- **Unit tests:** `TlsProvider` construction from valid/invalid configs, cert parsing, mtime change detection, cert expiry calculation, config validation edge cases
- **Integration tests:** Full TCP connect + auth + publish/subscribe over TLS, HTTP publish/query/fetch over HTTPS, mTLS accept/reject with various failure modes, cert reload mid-operation
- **Test certs:** Generated at test time via `rcgen` — no checked-in cert files. Helper function `generate_test_certs() -> (CertifiedKey, Certificate)` shared across test modules.
- **CI:** Tests run with TLS enabled AND disabled to ensure both paths work. Separate test matrix entries to avoid masking failures.
- **Regression:** All existing integration tests (non-TLS) must continue passing without modification. The stream genericization in Phase 16.1 step 4 is validated by running the full test suite before adding any TLS code.

## Risks and Mitigations

| Risk | Mitigation |
|---|---|
| ~~`axum-server` shutdown behavior differs from `axum::serve()`~~ | Resolved: using custom `TlsListener` with `axum::serve()` for both paths — identical shutdown code. No `axum-server` dependency. |
| `aws-lc-rs` build dependency adds C compiler requirement | Use `ring` backend feature flag if C compiler is not available in CI |
| TLS handshake adds latency to connection setup | Benchmark handshake overhead. 1-3ms typical for TLS 1.3 — acceptable. Log slow handshakes (>100ms). |
| Cert reload race: reload happens mid-handshake | `ArcSwap` is atomic — in-progress handshakes use the old acceptor, new handshakes get the new one. No race. |
| mTLS + existing auth tests: doubling test matrix | mTLS tests are additive — they don't replace API key auth tests. Keep test sets separate. |
| Generic `run_connection` may increase compile time | Two monomorphizations (`TcpStream`, `TlsStream<TcpStream>`) is negligible. Not a real risk. |

## Out of Scope (Future Work)

- **STARTTLS / upgrade-in-place** — the port is TLS or plaintext, not both. Simpler to reason about and operate.
- **SNI-based routing** — all connections use the same cert. Multi-tenant SNI would require per-namespace certs.
- **CRL/OCSP checking** — `rustls 0.23` supports CRLs via `WebPkiClientVerifier::builder().with_crls()`. Can be added as a follow-up without protocol changes.
- **Client-side TLS** (for SDK) — this plan covers server-side only. SDK TLS support would be documented separately.
- **Dual-port listening** (TLS + plaintext simultaneously) — useful for migration but adds operational complexity. Defer until requested.
