GPU Monitoring

Real-time GPU metrics collection with Andon alerting system for proactive issue detection.

Toyota Principle: Andon (行灯)

Visual signaling system that alerts operators to problems. GPU monitoring provides real-time visibility into hardware health with automatic alerts.

Quick Start

#![allow(unused)]
fn main() {
use entrenar::monitor::gpu::{GpuMonitor, GpuMetrics, AndonSystem};

// Create monitor
let monitor = GpuMonitor::new()?;

// Collect metrics
let metrics = monitor.collect_metrics()?;

for gpu in &metrics {
    println!("GPU {}: {}°C, {}% util, {:.1} GB / {:.1} GB",
        gpu.device_id,
        gpu.temperature_celsius,
        gpu.utilization_percent,
        gpu.memory_used_bytes as f64 / 1e9,
        gpu.memory_total_bytes as f64 / 1e9
    );
}

// Check for alerts
let andon = AndonSystem::default();
let alerts = andon.check(&metrics);

for alert in alerts {
    eprintln!("ALERT [severity {}]: {}", alert.severity(), alert.message());
}
}

GPU Metrics

#![allow(unused)]
fn main() {
use entrenar::monitor::gpu::GpuMetrics;

// Available metrics
pub struct GpuMetrics {
    pub device_id: u32,
    pub name: String,
    pub temperature_celsius: u32,
    pub utilization_percent: u32,
    pub memory_used_bytes: u64,
    pub memory_total_bytes: u64,
    pub power_watts: f32,
    pub power_limit_watts: f32,
}

// Derived metrics
let memory_percent = metrics.memory_percent();
let power_percent = metrics.power_percent();
}

Andon Alert System

#![allow(unused)]
fn main() {
use entrenar::monitor::gpu::{AndonSystem, GpuAlert, AlertConfig};

// Default thresholds
let andon = AndonSystem::default();

// Custom thresholds
let config = AlertConfig {
    thermal_threshold: 80,      // °C
    memory_threshold: 90,       // %
    power_threshold: 95,        // %
    idle_timeout_secs: 300,     // 5 minutes
};
let andon = AndonSystem::with_config(config);

// Check for alerts
let alerts = andon.check(&metrics);
}

Alert Types

#![allow(unused)]
fn main() {
use entrenar::monitor::gpu::GpuAlert;

// Thermal throttling
GpuAlert::ThermalThrottling {
    device: 0,
    temp: 85,
    threshold: 80,
}

// Memory pressure
GpuAlert::MemoryPressure {
    device: 0,
    used_percent: 95,
    threshold: 90,
}

// Power limit
GpuAlert::PowerLimit {
    device: 0,
    power_percent: 98,
    threshold: 95,
}

// GPU idle (possible hang)
GpuAlert::GpuIdle {
    device: 0,
    duration_secs: 600,
}
}

Continuous Monitoring

#![allow(unused)]
fn main() {
use std::time::Duration;
use std::thread;

let monitor = GpuMonitor::new()?;
let andon = AndonSystem::default();

loop {
    let metrics = monitor.collect_metrics()?;

    // Log metrics
    for gpu in &metrics {
        log_metrics(gpu);
    }

    // Check alerts
    let alerts = andon.check(&metrics);
    for alert in alerts {
        send_alert(&alert);
    }

    thread::sleep(Duration::from_secs(5));
}
}

Sparkline Visualization

#![allow(unused)]
fn main() {
use entrenar::monitor::gpu::sparkline;

// Create ASCII sparkline from values
let utilization_history = vec![45, 67, 82, 91, 88, 75, 60];
let spark = sparkline(&utilization_history);
println!("Utilization: {}", spark);  // ▃▅▆█▇▅▄
}

Integration with Training

#![allow(unused)]
fn main() {
use entrenar::train::{Trainer, TrainerConfig};
use entrenar::train::callback::GpuMonitorCallback;

let config = TrainerConfig::default();
let mut trainer = Trainer::new(config);

// Add GPU monitoring callback
trainer.add_callback(GpuMonitorCallback::new()
    .with_interval_secs(10)
    .with_thermal_threshold(80)
    .on_alert(|alert| {
        eprintln!("GPU Alert: {}", alert.message());
        // Optionally pause training
    }));

trainer.fit(&model, &dataset)?;
}

Prometheus Export

Export metrics for Prometheus scraping:

#![allow(unused)]
fn main() {
use entrenar::monitor::prometheus::PrometheusExporter;

let exporter = PrometheusExporter::new()
    .with_prefix("entrenar")
    .with_port(9090);

// Register GPU metrics
exporter.register_gpu_metrics(&monitor)?;

// Start HTTP server
exporter.start()?;
}

Metrics exposed:

entrenar_gpu_temperature_celsius{device="0"}
entrenar_gpu_utilization_percent{device="0"}
entrenar_gpu_memory_used_bytes{device="0"}
entrenar_gpu_power_watts{device="0"}

Cargo Run Example

# Monitor GPUs
cargo run --example gpu_monitor

# With custom interval
cargo run --example gpu_monitor -- --interval 5

# Export to Prometheus
cargo run --example gpu_monitor -- --prometheus --port 9090

Mock Backend for Testing

#![allow(unused)]
fn main() {
use entrenar::monitor::gpu::MockGpuBackend;

// Create mock metrics for testing
let mock = MockGpuBackend::new()
    .with_device(0, "Mock GPU 0")
    .with_temperature(75)
    .with_utilization(85)
    .with_memory(8 * 1024 * 1024 * 1024, 16 * 1024 * 1024 * 1024);

let metrics = mock.collect_metrics()?;
}

Best Practices

Set appropriate thresholds - Balance sensitivity vs noise
Monitor continuously - 5-10 second intervals recommended
Log metrics for analysis - Useful for post-mortem debugging
Integrate with alerting - PagerDuty, Slack, etc.
Use mock backend for tests - Don't require real GPUs

NVML Integration

GPU metrics are collected via NVIDIA Management Library (NVML):

# Verify NVML is available
nvidia-smi

# Required for GPU monitoring
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

Entrenar - Training & Optimization Library