# Patterns

Circuit breakers, registries, and budget tracking patterns.

## Circuit Breaker

Failure isolation pattern to prevent cascade failures.

### State Machine

```
┌─────────────────────────────────────────────────────────┐
│                  Circuit Breaker FSM                     │
├─────────────────────────────────────────────────────────┤
│                                                          │
│      ┌────────┐                              ┌────────┐ │
│      │ CLOSED │─── failures >= threshold ───▶│  OPEN  │ │
│      │        │                              │        │ │
│      │ Normal │                              │ Reject │ │
│      │ traffic│                              │  all   │ │
│      └────────┘                              └────────┘ │
│          ▲                                       │      │
│          │                              timeout  │      │
│          │                                       ▼      │
│          │                               ┌────────────┐ │
│          │                               │ HALF-OPEN  │ │
│          │                               │   Test     │ │
│          └─── success >= threshold ──────│  traffic   │ │
│                                          └────────────┘ │
│                                                 │       │
│                                   failure ──────┘       │
│                                   (back to OPEN)        │
│                                                          │
└──────────────────────────────────────────────────────────┘
```

### Basic Usage

```rust
use sentinel_common::{CircuitBreaker, CircuitBreakerConfig};

// Configure thresholds
let config = CircuitBreakerConfig {
    failure_threshold: 5,      // Open after 5 failures
    success_threshold: 2,      // Close after 2 successes
    timeout_seconds: 30,       // Wait 30s before half-open
    half_open_max_requests: 1, // Allow 1 request in half-open
};

let breaker = CircuitBreaker::new(config);
```

### Request Flow

```rust
async fn call_upstream(breaker: &CircuitBreaker) -> Result<Response, Error> {
    // Check if circuit is closed (requests allowed)
    if !breaker.is_closed() {
        return Err(Error::CircuitOpen);
    }

    // Make the call
    match upstream_request().await {
        Ok(response) => {
            breaker.record_success();
            Ok(response)
        }
        Err(e) => {
            breaker.record_failure();
            Err(e)
        }
    }
}
```

### State Inspection

```rust
use sentinel_common::CircuitBreakerState;

// Get current state
match breaker.state() {
    CircuitBreakerState::Closed => println!("Normal operation"),
    CircuitBreakerState::Open => println!("Circuit open, rejecting"),
    CircuitBreakerState::HalfOpen => println!("Testing recovery"),
}

// Get counters
let failures = breaker.consecutive_failures();
let successes = breaker.consecutive_successes();

// Manual reset
breaker.reset();
```

### Named Circuit Breakers

```rust
// Create with name for logging
let breaker = CircuitBreaker::with_name(config, "payment-gateway");
```

## Registry

Generic thread-safe component storage.

### Basic Usage

```rust
use sentinel_common::Registry;

// Create registry
let routes: Registry<Route> = Registry::new();

// Or with initial capacity
let routes: Registry<Route> = Registry::with_capacity(100);
```

### Operations

```rust
// Insert item
routes.insert("api-v1".to_string(), Arc::new(route)).await;

// Get item
if let Some(route) = routes.get("api-v1").await {
    // Use route
}

// Check existence
if routes.contains("api-v1").await {
    // Route exists
}

// Remove item
let removed = routes.remove("api-v1").await;

// Get all keys
let keys = routes.keys().await;

// Get count
let count = routes.len().await;
```

### Atomic Replacement (Hot Reload)

```rust
// Build new configuration
let mut new_routes = HashMap::new();
new_routes.insert("api-v1".to_string(), Arc::new(new_route));
new_routes.insert("api-v2".to_string(), Arc::new(new_route_v2));

// Atomic swap - returns old routes
let old_routes = routes.replace(new_routes).await;

// Old routes continue serving in-flight requests
// New routes used for new requests
```

### Snapshot

```rust
// Get a point-in-time snapshot
let snapshot = routes.snapshot().await;

// Iterate over snapshot
for (id, route) in &snapshot {
    println!("{}: {:?}", id, route);
}
```

### Custom Operations

```rust
// Execute with read lock
routes.with_read(|map| {
    // Read-only access to internal map
    map.len()
}).await;

// Execute with write lock
routes.with_write(|map| {
    // Mutable access to internal map
    map.retain(|k, _| k.starts_with("api-"));
}).await;
```

## ScopedRegistry

Hierarchical registry with scope-based resolution.

### Setup

```rust
use sentinel_common::{ScopedRegistry, QualifiedId, Scope};

let registry: ScopedRegistry<Policy> = ScopedRegistry::new();
```

### Inserting Items

```rust
// Insert at global scope
registry.insert(
    QualifiedId::global("default-timeout"),
    Arc::new(Policy::timeout(30000)),
).await;

// Insert at namespace scope
registry.insert(
    QualifiedId::namespaced("production", "timeout"),
    Arc::new(Policy::timeout(10000)),
).await;

// Insert at service scope
registry.insert(
    QualifiedId::in_service("production", "payments", "timeout"),
    Arc::new(Policy::timeout(5000)),
).await;

// Insert and export (visible from all scopes)
registry.insert_exported(
    QualifiedId::namespaced("production", "shared-auth"),
    Arc::new(Policy::auth()),
).await;
```

### Resolution

```rust
// Resolve from service scope
let scope = Scope::Service {
    namespace: "production".to_string(),
    service: "payments".to_string(),
};

// Finds "production:payments:timeout" (5000ms)
let policy = registry.resolve("timeout", &scope).await;

// Resolution chain:
// 1. production:payments:timeout (found!)
// 2. production:timeout
// 3. Exported names
// 4. timeout (global)
```

### Direct Lookup

```rust
// By canonical ID
let policy = registry.get_by_canonical("production:payments:timeout").await;

// By QualifiedId
let qid = QualifiedId::in_service("production", "payments", "timeout");
let policy = registry.get(&qid).await;
```

## Token Budgets

Usage limits for inference endpoints.

### Configuration

```rust
use sentinel_common::{TokenBudgetConfig, BudgetPeriod};

let config = TokenBudgetConfig {
    period: BudgetPeriod::Daily,
    limit: 1_000_000,                      // 1M tokens/day
    alert_thresholds: vec![0.80, 0.90, 0.95],
    enforce: true,                         // Block when exhausted
    rollover: false,                       // No rollover
    burst_allowance: Some(0.10),          // 10% burst allowed
};
```

### Budget Periods

```rust
use sentinel_common::BudgetPeriod;

// Predefined periods
let hourly = BudgetPeriod::Hourly;   // Resets every hour
let daily = BudgetPeriod::Daily;     // Resets at midnight UTC
let monthly = BudgetPeriod::Monthly; // Resets on 1st

// Custom period
let custom = BudgetPeriod::Custom { seconds: 3600 * 4 }; // 4 hours

// Get duration
let duration = daily.as_duration(); // 86400 seconds
```

### Checking Budget

```rust
use sentinel_common::BudgetCheckResult;

match tracker.check(estimated_tokens) {
    BudgetCheckResult::Allowed { remaining } => {
        println!("Allowed, {} tokens remaining", remaining);
    }
    BudgetCheckResult::Exhausted { retry_after_secs } => {
        println!("Budget exhausted, retry after {}s", retry_after_secs);
    }
    BudgetCheckResult::Soft { remaining, over_by } => {
        println!("Allowed via burst, {} over limit", over_by);
    }
}

// Check if allowed
if result.is_allowed() {
    // Process request
}
```

### Budget Status

```rust
use sentinel_common::TenantBudgetStatus;

let status = tracker.status();
println!("Used: {} / {}", status.tokens_used, status.tokens_limit);
println!("Remaining: {}", status.tokens_remaining);
println!("Usage: {:.1}%", status.usage_percent * 100.0);
println!("Exhausted: {}", status.exhausted);
println!("Period: {} to {}", status.period_start, status.period_end);
```

### Alerts

```rust
use sentinel_common::BudgetAlert;

// Alerts triggered when thresholds crossed
fn handle_alert(alert: BudgetAlert) {
    println!(
        "Budget alert for {}: {:.0}% used ({}/{})",
        alert.tenant,
        alert.usage_percent() * 100.0,
        alert.tokens_used,
        alert.tokens_limit,
    );
}
```

## Cost Attribution

Track costs for inference requests.

### Configuration

```rust
use sentinel_common::{CostAttributionConfig, ModelPricing};

let config = CostAttributionConfig {
    enabled: true,
    pricing: vec![
        ModelPricing {
            model_pattern: "gpt-4*".to_string(),
            input_cost_per_million: 30.0,
            output_cost_per_million: 60.0,
            currency: None, // Use default
        },
        ModelPricing {
            model_pattern: "gpt-3.5-turbo*".to_string(),
            input_cost_per_million: 0.5,
            output_cost_per_million: 1.5,
            currency: None,
        },
    ],
    default_input_cost: 1.0,
    default_output_cost: 2.0,
    currency: "USD".to_string(),
};
```

### Pattern Matching

```rust
let pricing = ModelPricing {
    model_pattern: "gpt-4*".to_string(),
    input_cost_per_million: 30.0,
    output_cost_per_million: 60.0,
    currency: None,
};

// Pattern matching
assert!(pricing.matches("gpt-4"));
assert!(pricing.matches("gpt-4-turbo"));
assert!(pricing.matches("gpt-4-0125-preview"));
assert!(!pricing.matches("gpt-3.5-turbo"));
```

### Cost Calculation

```rust
use sentinel_common::CostResult;

// Calculate cost
let cost = pricing.calculate_cost(1000, 500);
// Input: 1000 tokens * $30/M = $0.03
// Output: 500 tokens * $60/M = $0.03
// Total: $0.06

let result = CostResult::new(
    "gpt-4".to_string(),
    1000,  // input tokens
    500,   // output tokens
    0.03,  // input cost
    0.03,  // output cost
    "USD".to_string(),
);

println!("Total: ${:.4} {}", result.total_cost, result.currency);
```

## Inference Health Checks

Advanced health checks for LLM backends.

### Inference Probe

Send minimal completion request:

```rust
use sentinel_common::InferenceProbeConfig;

let config = InferenceProbeConfig {
    endpoint: "/v1/completions".to_string(),
    model: "gpt-3.5-turbo".to_string(),
    prompt: ".".to_string(),
    max_tokens: 1,
    timeout_secs: 30,
    max_latency_ms: Some(5000), // Mark unhealthy if > 5s
};
```

### Model Status

Query provider status endpoints:

```rust
use sentinel_common::ModelStatusConfig;

let config = ModelStatusConfig {
    endpoint_pattern: "/v1/models/{model}/status".to_string(),
    models: vec!["gpt-4".to_string(), "gpt-3.5-turbo".to_string()],
    expected_status: "ready".to_string(),
    status_field: "status".to_string(),
    timeout_secs: 30,
};
```

### Queue Depth

Monitor backend queue depth:

```rust
use sentinel_common::QueueDepthConfig;

let config = QueueDepthConfig {
    header: Some("X-Queue-Depth".to_string()),
    body_field: None,
    endpoint: None,
    degraded_threshold: 100,   // Mark degraded if > 100
    unhealthy_threshold: 500,  // Mark unhealthy if > 500
    timeout_secs: 30,
};
```

### Warmth Detection

Detect cold models:

```rust
use sentinel_common::{WarmthDetectionConfig, ColdModelAction};

let config = WarmthDetectionConfig {
    sample_size: 10,
    cold_threshold_multiplier: 2.0,  // 2x baseline = cold
    idle_cold_timeout_secs: 300,     // Cold after 5min idle
    cold_action: ColdModelAction::MarkDegraded,
};
```

**Cold Model Actions:**
- `LogOnly` - Just log (default)
- `MarkDegraded` - Lower weight in LB
- `MarkUnhealthy` - Exclude until warmed
