Real-Time Training Monitoring

The monitor module provides comprehensive real-time visibility into training runs, implementing Toyota Way principles for quality assurance.

Features

Quick Start

#![allow(unused)]
fn main() {
use entrenar::monitor::{MetricsCollector, Metric, Dashboard, HanseiAnalyzer};

// Create collector
let mut collector = MetricsCollector::new();

// During training loop
for epoch in 0..100 {
    let loss = train_epoch(&model, &data);
    let accuracy = evaluate(&model, &val_data);

    collector.record(Metric::Loss, loss);
    collector.record(Metric::Accuracy, accuracy);
    collector.record(Metric::Epoch, epoch as f64);
}

// Generate post-training report
let analyzer = HanseiAnalyzer::new();
let report = analyzer.analyze("my-training-run", &collector, duration_secs);
println!("{}", analyzer.format_report(&report));
}

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Training Loop                             │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐        │
│  │ Forward │→│ Backward │→│ Optimize │→│ Metrics │        │
│  └─────────┘  └─────────┘  └─────────┘  └────┬────┘        │
└───────────────────────────────────────────────┼─────────────┘
                                                │
                                                ▼
┌─────────────────────────────────────────────────────────────┐
│                  MetricsCollector                            │
│  ┌──────────────────────────────────────────────────────┐  │
│  │ Welford's Algorithm: O(1) per update                  │  │
│  │ - Running mean, variance, min, max                    │  │
│  │ - NaN/Inf detection                                   │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
         │              │              │              │
         ▼              ▼              ▼              ▼
    ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐
    │Dashboard│   │  Drift  │   │  Andon  │   │ Hansei  │
    │ (ASCII) │   │Detector │   │ System  │   │Analyzer │
    └─────────┘   └─────────┘   └─────────┘   └─────────┘

Performance

The monitor module is designed for minimal overhead:

Next Steps

• Metrics Collection - Recording training metrics
• Terminal Dashboard - Live visualization
• Drift Detection - Anomaly detection
• Andon Alerting - Stop-the-line on critical failures
• Hansei Reports - Post-training analysis