# Server Tuning Guide

Recommended configurations for three common deployment sizes. All values assume SSD storage, Linux (or macOS for dev), and a single fafb-q process per host.

These are starting points — monitor Prometheus metrics and adjust based on actual workload patterns.

## System Preparation (all tiers)

```bash
# File descriptor limit — set in /etc/security/limits.conf or systemd unit
ulimit -n 65536

# Recommended sysctl (Linux)
sysctl -w net.core.somaxconn=4096
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
```

---

## Tier 1 — 1 CPU / 1 GB RAM

Best for: dev/staging, low-traffic internal tools, < 50 clients.

**Memory budget:** ~600 MB usable after OS overhead.

```toml
[tcp]
bind_address = "0.0.0.0"
port = 9876
max_message_size = 262144            # 256 KB — keep frames small
max_connections = 500                # hard cap to protect memory
subscriber_channel_buffer = 128      # smaller per-subscriber buffer
slow_consumer_policy = "disconnect_slow"  # disconnect rather than buffer drops
slow_consumer_threshold = 50
slow_consumer_disconnect_after = 200
idle_timeout_secs = 15       # fast dead-client detection; clients must send Ping keepalive

[http]
bind_address = "0.0.0.0"
port = 8080

[storage]
data_dir = "./data"
fsync_strategy = "every_batch"
batch_flush_interval_ms = 200        # less frequent flushes to reduce I/O
batch_flush_size_bytes = 2097152     # 2 MB flush threshold
max_open_fds = 128                   # conserve FDs
max_segments_per_topic = 16          # aggressive segment cleanup
retention_check_interval_secs = 60
index_interval = 512                 # fewer index entries to save memory
segment_merge_threshold_bytes = 8388608   # 8 MB merge threshold
compaction_interval_secs = 600            # compact less often

[logging]
level = "info"
format = "text"
stdout = true
file_enabled = false                 # save disk I/O; use stdout → journald

[deploy]
drain_timeout_secs = 15
shutdown_timeout_secs = 5

[metrics]
enabled = true

[queue]
max_depth = 10000                    # 10k messages per topic
backpressure_policy = "DropOldest"
default_topic_max_bytes = 33554432   # 32 MB per topic
global_max_memory_bytes = 268435456  # 256 MB total queue memory
max_consumer_groups_per_topic = 100
cold_batch_size = 64                 # smaller cold reads
max_redelivery_attempts = 3
dlq_topic_suffix = "__dlq"

# Example namespace with tight quotas
[namespaces.default]
api_key = "change-me-in-production-key"
permissions = ["Subscribe", "Publish", "Query"]
max_topics = 50
max_total_messages = 100000
max_connections = 200
max_disk_bytes = 1073741824          # 1 GB disk quota

[namespaces.default.retention]
max_age_secs = 86400                 # 1 day
max_bytes = 536870912                # 512 MB per topic
max_segments = 16
```

**Expected capacity:**

| Metric | Estimate |
|--------|----------|
| Concurrent connections | 100–500 |
| Namespaces | 5–10 |
| Topics (total) | 20–50 |
| Publish throughput | 5k–15k msg/s |
| Delivery throughput | 5k–10k msg/s |
| In-memory messages | ~250k (at 1 KB avg) |

**Key constraints:** Single CPU means tokio runs 1 worker thread. All codec, I/O, and delivery fan-out share that core. Keep topic fan-out low (< 20 subscribers per topic) and message sizes small.

---

## Tier 2 — 2 CPU / 4 GB RAM

Best for: small production deployments, moderate internal messaging, < 500 clients.

**Memory budget:** ~3 GB usable after OS overhead.

```toml
[tcp]
bind_address = "0.0.0.0"
port = 9876
max_message_size = 524288            # 512 KB
max_connections = 3000
subscriber_channel_buffer = 512
slow_consumer_policy = "drop_for_slow"
slow_consumer_threshold = 100
slow_consumer_disconnect_after = 1000
idle_timeout_secs = 30       # clients must send Ping keepalive (recommended every 10s)

[http]
bind_address = "0.0.0.0"
port = 8080

[storage]
data_dir = "./data"
fsync_strategy = "every_batch"
batch_flush_interval_ms = 100
batch_flush_size_bytes = 4194304     # 4 MB
max_open_fds = 512
max_segments_per_topic = 32
retention_check_interval_secs = 60
index_interval = 256
segment_merge_threshold_bytes = 16777216  # 16 MB
compaction_interval_secs = 300

[logging]
level = "info"
format = "json"                      # structured for log aggregation
stdout = true
file_enabled = true
file_dir = "./logs"
file_prefix = "fafb-q"
rotation = "daily"
max_age_days = 7
max_total_size_mb = 100

[deploy]
drain_timeout_secs = 30
shutdown_timeout_secs = 10

[metrics]
enabled = true

[queue]
max_depth = 50000                    # 50k messages per topic
backpressure_policy = "DropOldest"
default_topic_max_bytes = 134217728  # 128 MB per topic
global_max_memory_bytes = 1610612736 # 1.5 GB total queue memory
max_consumer_groups_per_topic = 500
cold_batch_size = 128
max_redelivery_attempts = 5
dlq_topic_suffix = "__dlq"

[namespaces.default]
api_key = "change-me-in-production-key"
permissions = ["Subscribe", "Publish", "Query"]
max_topics = 500
max_total_messages = 500000
max_connections = 1000
max_disk_bytes = 10737418240         # 10 GB disk quota

[namespaces.default.retention]
max_age_secs = 259200                # 3 days
max_bytes = 5368709120               # 5 GB per topic
max_segments = 32
```

**Expected capacity:**

| Metric | Estimate |
|--------|----------|
| Concurrent connections | 500–3,000 |
| Namespaces | 10–50 |
| Topics (total) | 100–500 |
| Publish throughput | 30k–80k msg/s |
| Delivery throughput | 25k–60k msg/s |
| In-memory messages | ~1.5M (at 1 KB avg) |

**Key constraints:** 2 tokio worker threads. Disk I/O becomes relevant — with 200+ active topics, flush cycles can saturate a single SSD's IOPS. Consider `fsync_strategy = "os_managed"` if durability guarantees can be relaxed. Watch `fafb_fd_evictions_total` — if high, raise `max_open_fds`.

---

## Tier 3 — 4 CPU / 8 GB RAM

Best for: production workloads, moderate-to-high throughput, up to several thousand clients.

**Memory budget:** ~6.5 GB usable after OS overhead.

```toml
[tcp]
bind_address = "0.0.0.0"
port = 9876
max_message_size = 1048576           # 1 MB
max_connections = 10000
subscriber_channel_buffer = 1024
slow_consumer_policy = "drop_for_slow"
slow_consumer_threshold = 100
slow_consumer_disconnect_after = 1000
idle_timeout_secs = 45       # slightly longer to reduce keepalive overhead at scale; clients ping every 15s

[http]
bind_address = "0.0.0.0"
port = 8080

[storage]
data_dir = "./data"
fsync_strategy = "every_batch"
batch_flush_interval_ms = 100
batch_flush_size_bytes = 4194304     # 4 MB
max_open_fds = 1024
max_segments_per_topic = 64
retention_check_interval_secs = 60
index_interval = 256
segment_merge_threshold_bytes = 16777216  # 16 MB
compaction_interval_secs = 300

[logging]
level = "info"
format = "json"
stdout = true
file_enabled = true
file_dir = "./logs"
file_prefix = "fafb-q"
rotation = "daily"
max_age_days = 14
max_total_size_mb = 200

[deploy]
drain_timeout_secs = 30
shutdown_timeout_secs = 10

[metrics]
enabled = true

[queue]
max_depth = 100000                   # 100k messages per topic
backpressure_policy = "DropOldest"
default_topic_max_bytes = 268435456  # 256 MB per topic
global_max_memory_bytes = 4294967296 # 4 GB total queue memory
max_consumer_groups_per_topic = 1000
cold_batch_size = 256
max_redelivery_attempts = 5
dlq_topic_suffix = "__dlq"

[namespaces.default]
api_key = "change-me-in-production-key"
permissions = ["Subscribe", "Publish", "Query"]
max_topics = 2000
max_total_messages = 2000000
max_connections = 5000
max_disk_bytes = 53687091200         # 50 GB disk quota

[namespaces.default.retention]
max_age_secs = 604800                # 7 days
max_bytes = 10737418240              # 10 GB per topic
max_segments = 64
```

**Expected capacity:**

| Metric | Estimate |
|--------|----------|
| Concurrent connections | 2,000–10,000 |
| Namespaces | 20–100 |
| Topics (total) | 200–2,000 |
| Publish throughput | 100k–300k msg/s |
| Delivery throughput | 80k–200k msg/s |
| In-memory messages | ~4M (at 1 KB avg) |

**Key constraints:** 4 tokio workers give real parallelism. CPU becomes the bottleneck before memory at high throughput. High fan-out topics (>100 subscribers each) will saturate CPU — use consumer groups to reduce fan-out. Monitor `fafb_cold_batch_reads_total` — frequent cold reads indicate consumers are falling behind the in-memory window and you should increase `global_max_memory_bytes` or tune retention.

---

## Scaling Knobs by Symptom

| Symptom | Metric to watch | Knob to turn |
|---------|----------------|--------------|
| Publishes rejected | `fafb_publish_rejected_total` | Increase `global_max_memory_bytes`, `max_depth`, or `default_topic_max_bytes` |
| Slow consumers piling up | `fafb_messages_evicted_total` | Lower `slow_consumer_threshold`, switch to `disconnect_slow`, or add consumer groups |
| High FD eviction rate | `fafb_fd_evictions_total` | Increase `max_open_fds` and `ulimit -n` |
| Cold reads frequent | `fafb_cold_batch_reads_total` | Increase `global_max_memory_bytes` or tighten retention to keep working set in memory |
| Disk quota hits | `fafb_quota_rejected_total` | Increase `max_disk_bytes` or tighten `retention` policy |
| Startup replay slow | Startup logs | Reduce `max_segments_per_topic`, increase retention aggressiveness |
| Connection refused | `max_connections` limit | Raise `max_connections`, check `ulimit -n` |

## Message Size Guidance

All throughput estimates above assume 256 B – 1 KB average message size. Larger messages reduce throughput roughly proportionally:

| Avg message size | Throughput multiplier |
|-----------------|----------------------|
| 256 B | 1.5–2x above estimates |
| 1 KB | 1x (baseline) |
| 10 KB | 0.3–0.5x |
| 100 KB | 0.05–0.1x |

At 100 KB+ messages, network bandwidth (1 Gbps ≈ 125 MB/s ≈ 1,250 msg/s) often becomes the bottleneck before CPU or memory.
