# NNTP Proxy - GitHub Copilot Instructions

This document provides comprehensive guidance for understanding and working with the NNTP Proxy codebase. It is optimized for GitHub Copilot to provide better context-aware suggestions.

**Last Updated:** May 4, 2026  
**Version:** See `Cargo.toml`.  
**Rust Edition:** 2024  
**MSRV:** See `Cargo.toml`.  
**Development Toolchain:** See `rust-toolchain.toml`.

---

## Table of Contents

1. [Project Overview](#project-overview)
2. [Architecture Overview](#architecture-overview)
3. [Module Structure](#module-structure)
4. [Core Concepts](#core-concepts)
5. [Coding Standards](#coding-standards)
6. [Performance Patterns](#performance-patterns)
7. [Testing Guidelines](#testing-guidelines)
8. [Common Tasks](#common-tasks)

---

## Project Overview

### What is NNTP Proxy?

A high-performance NNTP (Network News Transfer Protocol) proxy server written in Rust with:
- **Hybrid routing mode** - Intelligent per-command routing that auto-switches to stateful when needed (default)
- **Health-aware load balancing** - Uses least-loaded or weighted round-robin backend selection
- **TLS/SSL support** - Secure outbound backend connections using rustls; client-side listening remains plain NNTP
- **Connection pooling** - Pre-authenticated connections with deadpool
- **Article caching** - Optional LRU cache for frequently accessed articles
- **Client authentication** - Config-based authentication with credential validation

### Design Philosophy

1. **Performance-first** - Lock-free routing, zero-allocation hot paths, SIMD-optimized parsing
2. **Type safety** - Newtype pattern for all domain values (Port, ServerName, MessageId, etc.)
3. **RFC compliance** - Strict adherence to RFC 3977 (NNTP), RFC 4643 (Auth), RFC 5536 (Message-ID)
4. **Testability** - 82%+ library code coverage, property-based testing, integration tests, mock servers
5. **Observability** - Structured logging with tracing, detailed metrics, error classification

### Runtime Binary

1. **`nntp-proxy`** - Main proxy server with three routing modes, built-in caching, and selectable dashboard/headless UI behavior

---

## Architecture Overview

### High-Level Flow

```
Client → Proxy (Routing Logic) → Backend Pool → Backend Server(s)
          ↓                          ↓
      Auth Handler              Connection Pool
          ↓                          ↓
    Session Manager            Health Checker
```

### Three Routing Modes

#### 1. **Hybrid Mode** (Default - Recommended)
- Starts in per-command routing for efficiency
- Auto-detects stateful commands (GROUP, XOVER, etc.)
- Seamlessly switches to dedicated backend connection
- **Best for:** Universal compatibility with optimal performance

#### 2. **Stateful Mode** (`--routing-mode stateful`)
- Traditional 1:1 client-to-backend mapping
- Each client gets dedicated backend connection
- Full NNTP protocol support
- **Best for:** Maximum compatibility, debugging, legacy deployments

#### 3. **Per-Command Mode** (`--routing-mode per-command`)
- Each command routes independently using the configured backend-selection strategy
- Rejects stateful commands (GROUP, NEXT, LAST, XOVER, etc.)
- Only supports message-ID based operations
- **Best for:** Indexing tools, specialized workloads

### Request Flow (Per-Command Mode)

```
1. Client connects → TcpStream
2. Proxy sends greeting (200 NNTP Proxy Ready)
3. Client sends command → Command Parser
4. Router selects backend (configured strategy)
5. Get connection from pool
6. Forward command → Backend
7. Stream response → Client (zero-copy)
8. Return connection to pool
9. Repeat from step 3
```

### Request Flow (Hybrid Mode)

```
1. Client connects → TcpStream
2. Proxy sends greeting
3. Start in per-command mode
4. For each command:
   a. Parse command (stateful vs stateless)
   b. If stateless → route per-command
   c. If stateful → switch to dedicated backend
5. Once switched, remain in stateful mode
```

---

## Module Structure

### Source Organization (`src/`)

```
src/
├── lib.rs                    # Public API exports and module declarations
├── bin/                      # Runtime entrypoints; public docs should describe a single nntp-proxy executable with selectable UI modes
├── args.rs                   # Shared CLI/env argument handling
├── auth/                     # Client and backend authentication
├── cache/                    # Memory, disk, and availability caching
├── command/                  # Command classification and handling
├── config/                   # Config types, defaults, loading, validation
├── metrics/                  # Metrics collection and persistence
├── pool/                     # Connection and buffer pools
├── protocol/                 # NNTP commands, responses, status parsing
├── proxy/                    # Builder and proxy lifecycle orchestration
├── router/                   # Backend selection strategies and queues
├── runtime.rs                # Shared runtime/bootstrap helpers
├── session/                  # Session state, handlers, streaming, retries
├── stream/                   # Plain/TLS/compressed connection stream types
├── tls.rs                    # TLS configuration and helpers
├── tui/                      # Terminal dashboard
└── types/ + types.rs         # Domain types and validated wrappers
```

### Test Organization (`tests/`)

```
tests/
├── integration_tests.rs      # End-to-end proxy behavior
├── proxy/                    # Config, routing, and proxy integration suites
├── rfc3977/                  # RFC 3977 compliance coverage
├── rfc4643/                  # RFC 4643 auth coverage
├── cache/                    # Cache-specific integration tests
├── test_helpers.rs           # Shared test utilities and mock servers
├── test_metrics.rs           # Metrics coverage
├── test_properties.rs        # Property-based tests
└── test_rfc_protocol_bugs.rs # Regression coverage for protocol edge cases
```

---

## Core Concepts

### 1. Type-Safe Domain Values

**Pattern:** All domain values use the newtype pattern with validation.

```rust
// ❌ WRONG - Primitive obsession
fn connect(host: String, port: u16) -> Result<()> { ... }

// ✅ CORRECT - Type-safe with validation
fn connect(host: HostName, port: Port) -> Result<()> { ... }
```

**Key Types:**
- `HostName` - Validated hostname (non-empty)
- `ServerName` - Validated server name (non-empty)
- `Port` - Validated port (1-65535)
- `MessageId<'a>` - RFC 5536 compliant message-ID with `<` and `@`
- `ClientId` - UUIDv4 for request tracking
- `BackendId` - Zero-cost wrapper around backend index
- `MaxConnections` - Non-zero connection limit
- `BufferSize` - Non-zero buffer size
- `CacheCapacity` - Non-zero cache capacity

**Benefits:**
- Compile-time validation
- Self-documenting code
- Impossible to mix up parameters
- Zero runtime cost (newtype pattern)

### 2. Zero-Allocation Hot Paths

**Critical Performance Path:** Command classification must be 4-6ns.

```rust
// classifier.rs - 70%+ of all NNTP traffic
impl NntpCommand {
    pub fn parse(command: &str) -> Self {
        // Direct byte comparisons, no allocations
        let bytes = command.as_bytes();
        
        // SIMD-friendly case matching
        if matches_any(cmd, ARTICLE_CASES) {
            return self.classify_article(bytes, cmd_end);
        }
        
        // ... ultra-fast classification
    }
}
```

**Rules:**
- No `String::new()` or `Vec::new()` in hot paths
- Use `&str` and `&[u8]` slices
- Prefer stack over heap
- Use `memchr` for fast byte searching (SIMD-accelerated)

### 3. Streaming Architecture

**Pattern:** Never buffer entire responses. Stream directly client ↔ backend.

```rust
// ❌ WRONG - Buffers entire response (memory explosion)
async fn forward_response(client: &mut TcpStream, backend: &mut TcpStream) {
    let mut response = Vec::new();
    backend.read_to_end(&mut response).await?;
    client.write_all(&response).await?;
}

// ✅ CORRECT - Streams with double-buffering
async fn stream_response(client: &mut WriteHalf, backend: &mut ReadHalf, buffer: &mut PooledBuffer) {
    loop {
        let n = backend.read(buffer.as_mut()).await?;
        if n == 0 { break; }
        client.write_all(&buffer[..n]).await?;
    }
}
```

**Benefits:**
- Constant memory usage regardless of article size
- 100x+ throughput improvement
- Handles GB-sized articles efficiently

### 4. Error Classification

**Pattern:** Distinguish error types for appropriate handling.

```rust
use crate::session::error_classification::ErrorClassifier;

match result {
    Err(e) if ErrorClassifier::is_client_disconnect(&e) => {
        debug!("Client disconnected (normal)");
    }
    Err(e) if ErrorClassifier::is_authentication_error(&e) => {
        error!("Auth failed: {}", e);
    }
    Err(e) => {
        warn!("Unexpected error: {}", e);
    }
}
```

**Error Categories:**
- **Client disconnect** - Broken pipe, connection reset (DEBUG level)
- **Authentication error** - Backend auth failure (ERROR level)
- **Network error** - Timeouts, DNS failures (WARN level)
- **Backend error** - Protocol violations, unexpected responses (ERROR level)

### 5. Connection Pool Management

**Pattern:** Use `deadpool` with custom health checking.

```rust
// Get connection from pool
let mut conn = provider.get_pooled_connection().await?;

// Use connection (automatically returned to pool on drop)
execute_command(&mut *conn, command).await?;

// Connection auto-returned unless error detected
```

**Pool Configuration:**
- `max_connections` - Per-backend connection limit (default: 10)
- Health checks - TCP peek + DATE command validation
- Prewarming - Create connections at startup
- Graceful shutdown - Close idle connections cleanly

### 6. Article Availability Tracking (CRITICAL - DO NOT MISUSE)

**Type:** `cache::ArticleAvailability` - Bitset tracking which backends have a specific article

**⚠️ DUAL PURPOSE - This type serves TWO critical functions:**

1. **Cache persistence** - Track article availability across requests (long-lived)
2. **430 retry tracking** - Track which backends tried during retry loop (transient, per-request)

**Pattern for 430 retry loops:**

```rust
// Create fresh availability tracker for THIS request
let mut availability = crate::cache::ArticleAvailability::new();

loop {
    let backend_id = router.route_command(client_id, command)?;
    
    // Check if we already tried this backend (it returned 430)
    if availability.has(backend_id) == Some(false) {
        // All backends exhausted - send 430 to client
        break;
    }
    
    // Execute command on backend
    let response = execute_on_backend(backend_id)?;
    
    if response.status_code() == 430 {
        // Mark this backend as NOT having the article
        availability.mark_missing(backend_id);
        continue; // Try next backend
    }
    
    // Success - return response
    return Ok(response);
}
```

**Understanding the bitset semantics:**
- `has(backend_id) == None` → Backend not yet tried, **attempt it**
- `has(backend_id) == Some(false)` → Backend tried and returned 430, **skip it**
- `has(backend_id) == Some(true)` → Backend has article (shouldn't happen in retry loop)

**CRITICAL - DO NOT:**
- ❌ Create a separate "tried backends" bitset - `ArticleAvailability` already does this!
- ❌ Use `mark_has()` to track "tried" - that means backend HAS the article
- ❌ Mark backends before trying them - mark AFTER getting the response
- ❌ Confuse the semantics - `mark_missing()` is for 430, `mark_has()` is for success

**WHY THIS WORKS:**
- The `checked` bitset tracks which backends we've tried
- The `has_article` bitset tracks the result (true=has, false=430)
- Combined: `has(backend_id) == Some(false)` means "tried and returned 430"
- This is exactly what we need - no duplicate tracking required

---

## Coding Standards

### Rust Code Style

1. **Follow Rust 2024 Edition idioms**
   - Use `const fn` where possible
   - Prefer `#[must_use]` on getter methods
   - Use `#[inline]` on hot path functions
   - Mark intentionally unused code with `#[allow(dead_code)]`

2. **Error Handling**
   ```rust
   // ✅ Use anyhow::Result for application code
   pub async fn handle_client(stream: TcpStream) -> Result<()> { ... }
   
   // ✅ Use thiserror for library errors
   #[derive(Debug, thiserror::Error)]
   pub enum ValidationError {
       #[error("Invalid hostname: {0}")]
       InvalidHostname(String),
   }
   
   // ✅ Use context for better error messages
   connection.read(&mut buf).await
       .context("Failed to read from backend")?;
   ```

3. **Async Patterns**
   ```rust
   // ✅ Use tokio::spawn for concurrent tasks
   tokio::spawn(async move {
       proxy.handle_client(stream, addr).await
   });
   
   // ✅ Use select! for concurrent operations with cancellation
   tokio::select! {
       result = client.read(&mut buf) => { ... }
       _ = shutdown_signal() => { ... }
   }
   
   // ❌ Don't block async runtime
   // std::thread::sleep(Duration::from_secs(1)); // WRONG!
   tokio::time::sleep(Duration::from_secs(1)).await; // CORRECT
   ```

4. **Naming Conventions**
   - **Modules:** `snake_case` (e.g., `connection_pool`)
   - **Types:** `PascalCase` (e.g., `ClientSession`)
   - **Functions:** `snake_case` (e.g., `route_command_sync`)
   - **Constants:** `SCREAMING_SNAKE_CASE` (e.g., `MAX_CONNECTIONS`)
   - **Type aliases:** `PascalCase` (e.g., `Result`)

5. **Documentation**
   ```rust
   /// Parse NNTP response status code
   ///
   /// Per [RFC 3977 §3.2](https://datatracker.ietf.org/doc/html/rfc3977#section-3.2),
   /// responses begin with a 3-digit status code.
   ///
   /// # Arguments
   /// * `data` - Raw response bytes
   ///
   /// # Returns
   /// Status code if valid, None otherwise
   ///
   /// # Examples
   /// ```
   /// assert_eq!(parse_status_code(b"200 OK"), Some(200));
   /// ```
   #[inline]
   pub fn parse_status_code(data: &[u8]) -> Option<u16> { ... }
   ```

   **Always include:**
   - RFC references for protocol-related code
   - Examples for public APIs
   - Performance characteristics for critical paths
   - Invariants and assumptions

### Performance Guidelines

#### 1. Hot Path Optimization

**Critical paths (<10ns target):**
- Command classification (`classifier.rs`)
- Response code parsing (`response.rs`)
- Backend selection (`router/mod.rs`)

**Rules:**
```rust
// ✅ Direct byte comparison (4-6ns)
if bytes[0..7] == *b"ARTICLE" { ... }

// ❌ String conversion (40-60ns)
if command.to_uppercase().starts_with("ARTICLE") { ... }

// ✅ memchr for byte searching (SIMD)
if let Some(pos) = memchr::memchr(b'<', bytes) { ... }

// ❌ Iterator-based search
if let Some(pos) = bytes.iter().position(|&b| b == b'<') { ... }
```

#### 2. Buffer Management

**Use `PooledBuffer` for I/O operations:**
```rust
// ✅ Get buffer from pool
let mut buffer = buffer_pool.get();

// Use buffer
let n = stream.read(buffer.as_mut()).await?;

// Buffer automatically returned to pool on drop
```

**Constants from `src/constants.rs`:**
```rust
use crate::constants::buffer::*;

const POOL: usize = 256 * 1024;        // 256KB pooled buffers
const COMMAND: usize = 512;            // Command buffer
const RESPONSE_MAX: usize = 1024 * 1024; // 1MB response limit
const STREAM_CHUNK: usize = 65536;     // 64KB streaming chunks
```

#### 3. Lock-Free Algorithms

**Router uses atomic operations:**
```rust
// ✅ Lock-free round-robin
let index = self.current_backend
    .fetch_add(1, Ordering::Relaxed) % self.backends.len();

// ✅ Atomic pending count tracking
backend.pending_count.fetch_add(1, Ordering::Relaxed);
```

**Never:**
- Use `Mutex` in hot paths
- Create contention on shared state
- Block on locks in async code

#### 4. Memory Allocation Patterns

```rust
// ✅ Pre-allocate with capacity
let mut backends = Vec::with_capacity(server_count);

// ✅ Reuse buffers across iterations
let mut line = String::with_capacity(COMMAND);
loop {
    line.clear();
    reader.read_line(&mut line).await?;
    // ... process line
}

// ❌ Allocate in loop
loop {
    let mut line = String::new(); // BAD: allocates every iteration
    reader.read_line(&mut line).await?;
}
```

#### 5. TCP Optimization

**Apply socket optimizations:**
```rust
use crate::network::{ConnectionOptimizer, NetworkOptimizer};

// Set large buffers for throughput
let optimizer = ConnectionOptimizer::new(&stream);
optimizer.optimize()?; // Sets 16MB buffers, TCP_NODELAY, etc.
```

**Constants from `src/constants.rs`:**
```rust
const HIGH_THROUGHPUT_RECV_BUFFER: usize = 16 * 1024 * 1024; // 16MB
const HIGH_THROUGHPUT_SEND_BUFFER: usize = 16 * 1024 * 1024; // 16MB
```

### Logging Standards

**Use structured logging with `tracing`:**

```rust
use tracing::{debug, info, warn, error};

// ✅ Structured fields
info!(
    client_addr = %client_addr,
    backend_id = ?backend_id,
    "Routing client to backend"
);

// ✅ Appropriate log levels
debug!("Got pooled connection");      // Internal state
info!("Client connected");            // Important events
warn!("Backend unhealthy");           // Degraded state
error!("Authentication failed");      // Errors requiring attention

// ❌ Don't log in hot paths
// info!("Processing command: {}", cmd); // BAD: Called millions of times

// ✅ Use debug for hot paths
debug!("Processing {}", cmd); // OK: Filtered out in production
```

**Log Level Guidelines:**
- `error!` - Auth failures, backend errors, unrecoverable errors
- `warn!` - Degraded performance, recoverable errors, unusual conditions
- `info!` - Connection lifecycle, routing decisions, important state changes
- `debug!` - Internal state, hot path activities (disabled in production)
- `trace!` - Extremely verbose debugging (disabled by default)

---

## Performance Patterns

### 1. Command Classification (4-6ns Hot Path)

**Location:** `src/command/classifier.rs`

**Pattern:** Ultra-fast byte-based classification with SIMD-friendly matching.

```rust
/// Classify NNTP command - PERFORMANCE CRITICAL (70%+ of traffic)
///
/// Target: 4-6ns on modern CPUs
/// - Zero allocations
/// - SIMD-friendly byte comparisons
/// - Branch prediction optimized (UPPERCASE first)
pub fn parse(command: &str) -> Self {
    let bytes = command.as_bytes();
    
    // Fast path: Check for message-ID (70%+ hit rate)
    if let Some(cmd_end) = find_command_end(bytes) {
        let cmd = &bytes[..cmd_end];
        
        // UPPERCASE checked first (95% of real traffic)
        if matches_any(cmd, ARTICLE_CASES) {
            return self.classify_article(bytes, cmd_end);
        }
        
        // ... more classifications
    }
    
    Self::Stateless // Unknown commands forwarded
}
```

**Key Techniques:**
- Pre-computed case arrays (UPPERCASE, lowercase, Titlecase)
- Direct byte comparison with `matches_any` macro
- Early returns for common cases
- Zero string allocations

### 2. Multiline Response Framing

**Location:** `src/session/multiline_framing.rs`

**Pattern:** all NNTP multiline response boundary handling goes through
`MultilineFramer`. Callers read into pooled buffers, pass those buffers to the
framer, and use the typed framed operation it returns. No other file should
inspect CR/LF/dot bytes, chunk ends, suffixes, leftovers, or terminator offsets
to decide response boundaries.

**Critical Points:**
- Never call `read_to_end()` or `read_to_string()` on proxy response paths.
- Never reimplement multiline terminator detection outside `MultilineFramer`.
- If a response is incomplete, feed the next backend buffer back into the same framer.
- After framing, use the typed framed response operation directly.
- Reuse `PooledBuffer` across reads and keep normal pass-through ARTICLE/BODY/HEAD data borrowed.

### 3. Connection Pool Prewarming

**Location:** `src/pool/prewarming.rs`

**Pattern:** Concurrent connection creation with batching.

```rust
/// Prewarm all connection pools concurrently
///
/// Creates connections in batches to avoid overwhelming backends
pub async fn prewarm_pools(
    providers: &[DeadpoolConnectionProvider],
    servers: &[Server],
) -> Result<()> {
    let tasks: Vec<_> = providers
        .iter()
        .zip(servers.iter())
        .map(|(provider, server)| {
            tokio::spawn(prewarm_single_pool(
                provider.clone(),
                server.clone(),
            ))
        })
        .collect();
    
    // Wait for all prewarming to complete
    for task in tasks {
        task.await??;
    }
    
    Ok(())
}
```

**Benefits:**
- Reduced first-request latency
- Predictable startup time
- Validates backend connectivity early

### 4. Health Check Strategy

**Location:** `src/pool/health_check.rs`

**Pattern:** Two-level health checking (TCP + Application).

```rust
/// Fast TCP-level check (non-blocking)
pub fn check_tcp_alive(conn: &mut ConnectionStream) -> RecycleResult<anyhow::Error> {
    match conn.try_read(&mut [0u8; 1]) {
        Ok(0) => Err(anyhow::anyhow!("Connection closed")),
        Ok(_) => Err(anyhow::anyhow!("Unexpected data")),
        Err(ref e) if e.kind() == ErrorKind::WouldBlock => Ok(()), // Healthy!
        Err(e) => Err(e.into()),
    }
}

/// Application-level check (sends DATE command)
pub async fn check_date_response(conn: &mut ConnectionStream) -> Result<(), HealthCheckError> {
    conn.write_all(DATE_COMMAND).await?;
    
    let mut buf = [0u8; HEALTH_CHECK_BUFFER_SIZE];
    let n = timeout(HEALTH_CHECK_TIMEOUT, conn.read(&mut buf)).await??;
    
    let response = std::str::from_utf8(&buf[..n])?;
    if !response.starts_with(EXPECTED_DATE_RESPONSE_PREFIX) {
        return Err(HealthCheckError::UnexpectedResponse(response.to_string()));
    }
    
    Ok(())
}
```

**Strategy:**
1. TCP check - Fast, catches closed connections
2. DATE command - Slower, validates NNTP protocol
3. Configurable intervals and thresholds

---

## Testing Guidelines

### Test Organization Rules

**CRITICAL: Where to put tests**

1. **Unit tests** - In the same file as the code being tested, inside `#[cfg(test)] mod tests { ... }`
   - Tests for a specific function/struct/module
   - No external dependencies or I/O
   - Fast, isolated, pure logic testing
   - Example: `src/protocol/response.rs` has `#[cfg(test)] mod tests` at the bottom

2. **Integration tests** - In `tests/*.rs` directory at repository root
   - Tests that involve multiple modules
   - Tests with I/O, networking, file system
   - Tests requiring mock servers or external resources
   - Example: `tests/integration_tests.rs`, `tests/test_auth_backend.rs`

3. **❌ NEVER create test submodules in `src/`**
   - DO NOT create `src/foo/tests/` directories
   - DO NOT create `src/foo/tests/mod.rs` or `src/foo/tests/test_*.rs` files
   - Unit tests belong inline in the module file itself

**Examples:**

```rust
// ✅ CORRECT - Unit tests in same file
// src/protocol/response.rs
pub struct StatusCode(u16);

impl StatusCode {
    pub fn parse(data: &[u8]) -> Option<Self> { ... }
}

#[cfg(test)]
mod tests {
    use super::*;
    
    #[test]
    fn test_parse_valid_code() {
        assert_eq!(StatusCode::parse(b"200"), Some(StatusCode(200)));
    }
}
```

```rust
// ✅ CORRECT - Integration tests in tests/ directory
// tests/test_proxy_routing.rs
use nntp_proxy::*;

#[tokio::test]
async fn test_end_to_end_routing() {
    // Multi-module integration test
}
```

```rust
// ❌ WRONG - Don't create test subdirectories in src/
// src/protocol/response/tests/mod.rs  ← NEVER DO THIS
// src/protocol/response/tests/parsing.rs  ← NEVER DO THIS
```

### Test Structure

The codebase uses a mix of unit tests (in module files) and integration tests (in `tests/` directory).

**Coverage target:** 74%+ overall, 90%+ for critical paths

#### Unit Tests (in `src/**/*.rs`)

```rust
#[cfg(test)]
mod tests {
    use super::*;
    
    #[test]
    fn test_basic_functionality() {
        let result = function_under_test();
        assert_eq!(result, expected_value);
    }
    
    #[tokio::test]
    async fn test_async_functionality() {
        let result = async_function().await;
        assert!(result.is_ok());
    }
}
```

#### Integration Tests (in `tests/*.rs`)

```rust
// tests/integration_tests.rs
use nntp_proxy::*;
use tokio::net::TcpListener;

#[tokio::test]
async fn test_proxy_routing() -> Result<()> {
    // 1. Setup mock backend
    let mock_port = find_available_port().await;
    tokio::spawn(create_mock_server(mock_port));
    
    // 2. Create proxy
    let proxy = NntpProxy::new(test_config(), RoutingMode::Hybrid)?;
    
    // 3. Test interaction
    let mut client = TcpStream::connect("127.0.0.1:8119").await?;
    // ... assertions
    
    Ok(()))
}
```

### Test Helpers

**Location:** `tests/test_helpers.rs`

```rust
/// Create a mock NNTP server for testing
/// Returns AbortHandle for automatic cleanup when dropped
pub fn spawn_mock_server(port: u16, server_name: &str) -> AbortHandle {
    MockNntpServer::new(port).with_name(server_name).spawn()
}

/// Get a random available port for testing (parallel execution safe)
pub async fn get_random_port() -> u16 {
    let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
    let port = listener.local_addr().unwrap().port();
    drop(listener); // Release port for mock server
    port
}

/// Create test configuration
pub fn create_test_config() -> Config {
    use crate::config::Server;
    use crate::types::Port;
    
    Config {
        servers: vec![
            Server::builder("localhost", Port::try_new(119).unwrap())
                .name("Test Server")
                .build()
                .unwrap()
        ],
        ..Default::default()
    }
}
```

**Important:** Always use random ports (port 0) in tests to avoid conflicts during parallel execution with `cargo nextest run`. Never hardcode port numbers.

### Testing Patterns

#### 1. Testing Command Classification

```rust
#[test]
fn test_article_by_message_id() {
    let cmd = NntpCommand::parse("ARTICLE <test@example.com>");
    assert_eq!(cmd, NntpCommand::ArticleByMessageId);
    assert!(!cmd.is_stateful());
}

#[test]
fn test_stateful_commands() {
    let commands = vec!["GROUP alt.test", "NEXT", "LAST", "XOVER 1-100"];
    for cmd in commands {
        let classified = NntpCommand::parse(cmd);
        assert!(classified.is_stateful(), "Command '{}' should be stateful", cmd);
    }
}
```

#### 2. Testing Response Parsing

```rust
#[test]
fn test_multiline_detection() {
    assert!(NntpResponse::is_multiline_response(220)); // ARTICLE
    assert!(NntpResponse::is_multiline_response(215)); // LIST
    assert!(!NntpResponse::is_multiline_response(200)); // Greeting
}

// Boundary, suffix, and raw terminator-offset tests belong in
// src/session/multiline_framing.rs. Tests elsewhere should exercise typed
// framed responses instead of duplicating CR/LF/dot checks.
```

#### 3. Testing Authentication

```rust
#[tokio::test]
async fn test_auth_required() {
    let auth_handler = AuthHandler::new(
        Some("testuser".to_string()),
        Some("testpass".to_string()),
    );
    
    assert!(auth_handler.is_enabled());
    assert!(auth_handler.validate_credentials("testuser", "testpass"));
    assert!(!auth_handler.validate_credentials("wrong", "wrong"));
}
```

#### 4. Property-Based Testing

**Critical for newtype wrappers and domain types**. Use `proptest` to generate 100+ random test cases per property.

```rust
use proptest::prelude::*;

// Test newtype invariants
proptest! {
    #[test]
    fn port_rejects_zero(port in 0u16..=0) {
        assert!(Port::new(port).is_err());
    }
    
    #[test]
    fn port_accepts_valid_range(port in 1u16..=65535) {
        let result = Port::new(port);
        assert!(result.is_ok());
        assert_eq!(result.unwrap().get(), port);
    }
    
    #[test]
    fn hostname_rejects_empty(s in "\\s*") {
        assert!(HostName::new(s).is_err());
    }
}

// Integration property tests
proptest! {
    #![proptest_config(ProptestConfig::with_cases(20))]
    
    #[test]
    fn prop_message_id_parsing(
        local in "[a-zA-Z0-9]{1,20}",
        domain in "[a-zA-Z0-9]{1,20}\\.[a-z]{2,5}"
    ) {
        let msg_id = format!("<{}@{}>", local, domain);
        // Verify parser handles all valid message IDs
    }
}
```

**Test Macros** (for common newtype patterns):

```rust
// src/test_macros.rs - Reusable test generation
test_nonzero_newtype_full!(BufferSize, 1, 1024, 65536);

// Expands to:
// - Zero rejection test
// - Valid range test
// - Ordering tests
// - Display/Debug tests
```

**Benefits:**
- 100+ random cases vs single hardcoded value
- Catches edge cases (0, MAX, overflow)
- Reduces test code by 26% (571 lines in this project)
- Massive thoroughness increase (10,000+ test cases)

#### 5. Testing Connection Pool

```rust
#[tokio::test]
async fn test_pool_prewarming() {
    let provider = create_test_provider();
    prewarm_single_pool(&provider, &test_config()).await?;
    
    let status = provider.status();
    assert_eq!(status.available, status.max_size);
}
```

#### 6. Testing Routing

```rust
#[test]
fn test_round_robin_selection() {
    let mut selector = BackendSelector::new();
    selector.add_backend(BackendId::from_index(0), "server1", provider1);
    selector.add_backend(BackendId::from_index(1), "server2", provider2);
    
    // Should alternate
    let backend1 = selector.route_command_sync(ClientId::new(), "LIST")?;
    let backend2 = selector.route_command_sync(ClientId::new(), "LIST")?;
    let backend3 = selector.route_command_sync(ClientId::new(), "LIST")?;
    
    assert_eq!(backend1.as_index(), 0);
    assert_eq!(backend2.as_index(), 1);
    assert_eq!(backend3.as_index(), 0); // Wraps around
}
```

### Mock Servers

**Smart Mock Server** (for integration tests):

```rust
pub async fn create_smart_mock_server(port: u16, name: &str) -> JoinHandle<()> {
    tokio::spawn(async move {
        let listener = TcpListener::bind(format!("127.0.0.1:{}", port)).await?;
        
        loop {
            let (mut stream, _) = listener.accept().await?;
            
            // Send greeting
            stream.write_all(b"200 Ready\r\n").await?;
            
            let mut buf = [0; 1024];
            loop {
                let n = stream.read(&mut buf).await?;
                let command = std::str::from_utf8(&buf[..n])?;
                
                // Handle specific commands
                if command.starts_with("GROUP") {
                    stream.write_all(b"211 100 1 100 alt.test\r\n").await?;
                } else if command.starts_with("ARTICLE") {
                    stream.write_all(b"220 0 <msg@test>\r\n").await?;
                    stream.write_all(b"Subject: Test\r\n\r\nBody\r\n.\r\n").await?;
                } else if command.starts_with("QUIT") {
                    stream.write_all(b"205 Goodbye\r\n").await?;
                    break;
                } else {
                    stream.write_all(b"200 OK\r\n").await?;
                }
            }
        }
    })
}
```

### Running Tests

```bash
# Run all tests with nextest (REQUIRED - 7x faster with parallel execution)
cargo nextest run

# Run with output
cargo nextest run --no-capture

# Run specific test
cargo nextest run -E 'test(hybrid_mode)'

# Run integration tests only
cargo nextest run --test integration_tests

# Run with coverage (requires cargo-llvm-cov)
cargo llvm-cov --html
```

**IMPORTANT:** Always use `cargo nextest run` for this project - it's ~7x faster than `cargo test` due to parallel execution. All tests use random ports to avoid conflicts during parallel runs.

**DO NOT use `cargo test`** - it's significantly slower and should be avoided. Use `cargo nextest run` instead.

**CRITICAL: When debugging tests or analyzing test output:**

❌ **DO NOT** use `cargo nextest` when you need to see debug output (`eprintln!`, `println!`, etc.)
- Nextest buffers and mangles output, making debug traces hard to read
- Use `cargo test` instead for debugging sessions

❌ **DO NOT** pipe test output through `grep`, `head`, or `tail` when debugging
- These commands hide critical debug output and context
- You WILL miss important information about test failures
- Run tests without pipes to see full output

✅ **DO** use these patterns for debugging:
```bash
# For seeing debug output - use cargo test with --nocapture
cargo test --test test_430_retry test_name -- --nocapture

# For comprehensive test output - no pipes
cargo test --test test_name -- --nocapture 2>&1

# For multiple tests with output
cargo test --test test_file -- --nocapture
```

✅ **DO** use nextest for:
- CI/CD pipelines
- Running full test suites quickly
- Tests that don't require debug output analysis
- Coverage runs

### Benchmarking

**Location:** `benches/`

```rust
// benches/command_parsing.rs
use divan::Bencher;

#[divan::bench]
fn classify_article_command(bencher: Bencher) {
    bencher.bench(|| {
        NntpCommand::parse("ARTICLE <msg@example.com>")
    });
}

#[divan::bench]
fn parse_status_code(bencher: Bencher) {
    bencher.bench(|| {
        NntpResponse::parse_status_code(b"200 OK")
    });
}
```

Run benchmarks:
```bash
cargo bench
```

---

## Common Tasks

### CRITICAL: Tools to NEVER use

**FORBIDDEN tools that waste time and tokens:**
- ❌ `sed` - Causes regex errors, incorrect substitutions, broken syntax
- ❌ `grep` - Use semantic_search or read_file instead
- ❌ `head` / `tail` - Truncates critical output, use full reads
- ❌ `perl` - Same issues as sed, plus availability problems
- ❌ `python` one-liners - Fragile, breaks on edge cases

**Why these fail:**
- Line-oriented tools don't understand code structure
- Regex edits repeatedly cause syntax errors and broken builds
- Output truncation hides critical errors and context
- They waste hours debugging what should be simple edits

**Use instead:**
- ✅ `replace_string_in_file` - Context-aware, validates syntax
- ✅ `multi_replace_string_in_file` - Batch edits efficiently
- ✅ `read_file` - Full file context, no truncation
- ✅ `semantic_search` - Understands code structure
- ✅ `run_in_terminal` - But NEVER pipe through head/tail/grep when debugging

### Important note about terminal commands

**DO NOT** interrupt long-running commands (Ctrl-C) - they often complete successfully if you wait.
Let commands finish naturally, especially:
- `cargo check` / `cargo build` - compilation takes time
- `cargo test` / `cargo nextest run` - test suites run fully
- File operations - I/O can be slow on some systems


### 1. Adding a New Backend Server

**In `config.toml`:**
```toml
[[servers]]
host = "news.example.com"
port = 119
name = "Example Server"
max_connections = 10
use_tls = false
```

**Programmatically:**
```rust
use nntp_proxy::config::Server;
use nntp_proxy::types::{MaxConnections, Port};

let server = Server::builder("news.example.com", Port::try_new(119).unwrap())
    .name("Example Server")
    .max_connections(MaxConnections::try_new(15).unwrap())
    .use_tls(true)
    .tls_verify_cert(true)
    .build()?;
```

### 2. Adding TLS Support to a Backend

```toml
[[servers]]
host = "secure.example.com"
port = 563
name = "Secure Server"
use_tls = true
tls_verify_cert = true
# Optional custom CA cert
# tls_cert_path = "/path/to/ca.pem"
```

### 3. Implementing a New NNTP Command

**1. Add to command classifier:**

```rust
// src/command/classifier.rs

// Add case array
command_cases!(
    NEWCMD_CASES,
    "NEWCMD",
    "newcmd",
    "Newcmd",
    "[RFC XXXX §X.X](url) - NEWCMD command\nDescription"
);

// Add to classify() function
pub fn parse(command: &str) -> Self {
    // ... existing code
    
    if matches_any(cmd, NEWCMD_CASES) {
        return Self::NewCommand;
    }
    
    // ...
}

// Add variant
pub enum NntpCommand {
    // ... existing variants
    NewCommand,
}
```

**2. Add command handler:**

```rust
// src/command/handler.rs

impl CommandHandler {
    pub fn handle_command(command: &str) -> CommandAction {
        match NntpCommand::parse(command) {
            // ... existing cases
            NntpCommand::NewCommand => CommandAction::ForwardStateless,
        }
    }
}
```

**3. Add tests:**

```rust
#[test]
fn test_new_command_classification() {
    assert_eq!(
        NntpCommand::parse("NEWCMD"),
        NntpCommand::NewCommand
    );
}
```

### 4. Adding Client Authentication

**In `config.toml`:**
```toml
[client_auth]
greeting = "200 NNTP proxy ready"

[[client_auth.users]]
username = "proxyuser"
password = "securepassword"
```

**Programmatically:**
```rust
let config = Config {
    client_auth: ClientAuth {
        greeting: None,
        users: vec![UserCredentials {
            username: "proxyuser".to_string(),
            password: "securepassword".to_string(),
        }],
    },
    ..Default::default()
};
```

### 5. Enabling Article Caching

**Caching is built into the main binaries:**

```bash
./target/release/nntp-proxy \
    --config cache-config.toml \
    --article-cache-capacity 256mb \
    --article-cache-ttl 3600 \
    --store-article-bodies true
```

**Configuration:**
```toml
# cache-config.toml
[cache]
article_cache_capacity = "256mb"
article_cache_ttl_secs = 3600
store_article_bodies = true
```

### 6. Debugging Connection Issues

**Enable verbose logging:**
```bash
RUST_LOG=debug ./target/release/nntp-proxy
```

**Check backend health:**
```rust
// Get pool status
let status = provider.status();
info!("Pool: {}/{} available, {} created", 
    status.available, status.max_size, status.created);

// Check backend load
if let Some(load) = router.backend_load(backend_id) {
    info!("Backend {:?} pending: {}", backend_id, load);
}
```

### 7. Monitoring Performance

**Key metrics to track:**

```rust
// Connection metrics
let (sent_bytes, recv_bytes) = session.handle_client(stream, addr).await?;
info!("Session: sent={}, received={}", sent_bytes, recv_bytes);

// Pool metrics
let status = provider.status();
let utilization = (status.max_size - status.available) as f64 / status.max_size as f64;
info!("Pool utilization: {:.1}%", utilization * 100.0);

// Router metrics
let load = router.backend_load(backend_id)?;
info!("Backend load: {} pending commands", load);
```

### 8. Detecting Terminal Output

**Check if running in interactive terminal:**

```rust
use std::io::IsTerminal;

if std::io::stdout().is_terminal() {
    // Interactive terminal - use colors, progress bars
    println!("✓ Running in interactive mode");
} else {
    // Piped/redirected - output machine-readable format
    println!("Running in non-interactive mode");
}

// Check stderr separately
if std::io::stderr().is_terminal() {
    eprintln!("\x1b[31mError\x1b[0m"); // Red color
} else {
    eprintln!("Error"); // Plain text
}
```

**Use cases:**
- Colored output for terminals only
- Progress bars in interactive mode
- JSON output when piped
- Different logging formats

### 9. Creating a Release Build

```bash
# Standard release
cargo build --release

# Optimized for size
cargo build --release --config profile.release.opt-level=\"z\"

# Cross-compile for other targets
cargo build --release --target x86_64-unknown-linux-gnu
cargo build --release --target aarch64-unknown-linux-gnu

# Create distribution package
./scripts/build-release.sh
```

### 10. Running with Systemd

**Service file (`nntp-proxy.service`):**

```ini
[Unit]
Description=NNTP Proxy Server
After=network.target

[Service]
Type=simple
User=nntp-proxy
Group=nntp-proxy
ExecStart=/usr/local/bin/nntp-proxy --config /etc/nntp-proxy/config.toml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

**Install and start:**
```bash
sudo systemctl enable nntp-proxy
sudo systemctl start nntp-proxy
sudo systemctl status nntp-proxy
```

---

## Troubleshooting

### Common Issues

**1. "No backends available for routing"**
- Check config.toml has servers defined
- Verify backend servers are reachable
- Check connection pool status

**2. "Failed to get pooled connection"**
- Pool exhausted - increase `max_connections`
- Backend unhealthy - check health checks
- Network issues - verify connectivity

**3. "Command not supported"**
- Using stateful command in per-command mode
- Switch to hybrid or stateful mode
- Or use message-ID based commands

**4. "Authentication failed"**
- Check backend credentials in config
- Verify backend supports AUTHINFO
- Check backend server logs

**5. High CPU usage**
- Too many worker threads - reduce with `--threads`
- Inefficient command classification - check logs
- Network issues causing retries

**6. Memory growth**
- Connection pool not releasing - check for panics
- Buffer pool exhaustion - increase pool size
- Cache too large - reduce cache capacity

---

## Additional Resources

### RFC References

- **[RFC 3977](https://datatracker.ietf.org/doc/html/rfc3977)** - NNTP Protocol
- **[RFC 4643](https://datatracker.ietf.org/doc/html/rfc4643)** - NNTP Authentication
- **[RFC 5536](https://datatracker.ietf.org/doc/html/rfc5536)** - Message-ID Format
- **[RFC 2980](https://datatracker.ietf.org/doc/html/rfc2980)** - Common NNTP Extensions (legacy)

### Code Organization Principles

1. **Separation of Concerns** - Each module has single responsibility
2. **Type Safety** - Newtype pattern for domain values
3. **Zero-Cost Abstractions** - No runtime overhead for safety
4. **Explicit over Implicit** - Clear, self-documenting code
5. **Performance by Default** - Hot paths optimized from the start

### Development Workflow

1. **Feature branches** - Create from `main`
2. **Tests first** - Write tests before implementation
3. **Benchmarks** - Profile before and after optimization
4. **Documentation** - Update docs with code changes
5. **Coverage** - Maintain 74%+ test coverage
6. **Pre-commit checks** - Always run before committing:
   ```bash
   cargo fmt
   cargo clippy --all-targets --all-features
   cargo nextest run
   ```

### Pre-1.0 Development

**IMPORTANT:** This project has NOT reached 1.0 yet and is a binary application, not a library.

**Do NOT:**
- Keep deprecated methods/functions around "for backwards compatibility"
- Add `#[deprecated]` attributes - just delete old code
- Maintain multiple versions of the same functionality
- Preserve old APIs - refactor directly

**DO:**
- Delete unused code immediately
- Rename methods directly without deprecation warnings
- Refactor aggressively - no need for stability guarantees
- Keep only one way to do things

**Rationale:** We're building toward 1.0, not maintaining a stable API. Clean, focused code is more important than backwards compatibility. Once we hit 1.0, we'll adopt proper deprecation practices.

**Note:** This project has Git pre-commit hooks that automatically run `cargo fmt` and `cargo clippy --all-targets --all-features`. If clippy finds issues or formatting is incorrect, the commit will be rejected. Fix all issues before committing.

**Checking compilation:** Use `cargo check` for fast compilation checking without generating binaries. Use `cargo build` only when you need the actual binary.

### NixOS Development Environment

**⚠️ NEVER use `cd` commands in terminal operations on NixOS**

This project uses `direnv` to automatically manage the Nix development environment. When you navigate to the project directory, `direnv` automatically:
- Loads the Nix flake development shell
- Sets up all required dependencies (Rust toolchain, build tools, etc.)
- Configures environment variables
- Ensures reproducible builds

**Why this matters:**
- `cd` commands in automated scripts/tools bypass direnv's environment activation
- This can cause builds to fail due to missing dependencies
- The environment is only properly activated when the shell itself changes directory
- Terminal tools should rely on the already-activated environment, not try to navigate

**Correct approach:**
```bash
# ✅ CORRECT - Run commands in current directory (direnv already active)
cargo build
cargo nextest run

# ❌ WRONG - Don't try to cd in scripts/automation
cd /home/mjc/projects/nntp-proxy && cargo build  # Bypasses direnv!
```

**For manual work:**
- Simply navigate to the project directory in your shell
- `direnv` will automatically activate (you'll see: "direnv: loading...")
- All tools (cargo, rustc, etc.) are then available
- Stay in the project directory for all development tasks

**If direnv isn't activating:**
```bash
# Allow direnv for this directory (one-time setup)
direnv allow

# Force reload if needed
direnv reload
```

---

**End of Instructions**

For more information, see:
- `README.md` - User documentation
- `CHANGELOG.md` - Version history
- `CACHE-PROXY.md` - Caching proxy documentation
- `TCP_TUNING.md` - Performance tuning guide
