Preflight Validation (Jidoka) - Entrenar - Training & Optimization Library
Description: A comprehensive guide to building neural network training systems with autograd, optimizers, LoRA/QLoRA, and quantization using EXTREME TDD methodology

# Preflight Validation

Catch data and environment issues before training starts. Research shows preflight validation prevents 30-50% of ML pipeline failures.

## Toyota Principle: Jidoka (自働化)

Built-in quality through automatic defect detection at source. Stop the line immediately when problems are detected.

## Quick Start

    #![allow(unused)]
    fn main() {
    use entrenar::storage::{Preflight, PreflightCheck, PreflightContext};

    // Standard data integrity checks
    let preflight = Preflight::standard();

    // Run checks
    let results = preflight.run(&data);

    if results.all_passed() {
        println!("All preflight checks passed!");
    } else {
        eprintln!("{}", results.report());
        std::process::exit(1);
    }
    }

## Built-in Checks

### Data Integrity Checks

    #![allow(unused)]
    fn main() {
    // No NaN values
    PreflightCheck::no_nan_values()

    // No infinite values
    PreflightCheck::no_inf_values()

    // Minimum sample count
    PreflightCheck::min_samples(1000)

    // Minimum feature count
    PreflightCheck::min_features(10)

    // Consistent dimensions (all rows same length)
    PreflightCheck::consistent_dimensions()

    // No constant features (zero variance)
    PreflightCheck::no_constant_features()

    // Class imbalance check
    PreflightCheck::label_balance(5.0)  // max 5:1 ratio
    }

### Environment Checks

    #![allow(unused)]
    fn main() {
    // Disk space check
    PreflightCheck::disk_space_mb(10240)  // 10GB minimum

    // Memory check
    PreflightCheck::memory_mb(8192)  // 8GB minimum

    // GPU availability
    PreflightCheck::gpu_available()
    }

## Preflight Presets

    #![allow(unused)]
    fn main() {
    // Standard data checks (NaN, Inf, dimensions, constant features)
    let preflight = Preflight::standard();

    // Comprehensive (data + environment)
    let preflight = Preflight::comprehensive();

    // Custom combination
    let preflight = Preflight::new()
        .add_check(PreflightCheck::no_nan_values())
        .add_check(PreflightCheck::min_samples(500))
        .add_check(PreflightCheck::disk_space_mb(5120));
    }

## Check Results

    #![allow(unused)]
    fn main() {
    let results = preflight.run(&data);

    // Overall status
    println!("Passed: {}", results.all_passed());

    // Counts
    println!("Passed: {}", results.passed_count());
    println!("Failed: {}", results.failed_count());
    println!("Warnings: {}", results.warning_count());
    println!("Skipped: {}", results.skipped_count());

    // Get failed checks
    for (check, result) in results.failed_checks() {
        eprintln!("FAILED: {} - {:?}", check.name, result);
    }

    // Get warnings
    for (check, result) in results.warnings() {
        println!("WARNING: {} - {:?}", check.name, result);
    }

    // Formatted report
    println!("{}", results.report());
    }

## Result Types

    #![allow(unused)]
    fn main() {
    use entrenar::storage::CheckResult;

    // Passed
    CheckResult::passed("All values valid")

    // Failed with details
    CheckResult::failed_with_details(
        "Found NaN values",
        "First locations: (0, 3), (5, 7)"
    )

    // Warning (non-fatal)
    CheckResult::warning("High class imbalance detected")

    // Skipped
    CheckResult::skipped("No data to check")
    }

## Optional vs Required Checks

    #![allow(unused)]
    fn main() {
    // Required check (blocks training if failed)
    PreflightCheck::no_nan_values()

    // Optional check (warning only)
    PreflightCheck::no_constant_features().optional()
    PreflightCheck::label_balance(5.0).optional()
    }

## Context-Based Thresholds

    #![allow(unused)]
    fn main() {
    use entrenar::storage::PreflightContext;

    let ctx = PreflightContext::new()
        .with_min_samples(10000)
        .with_min_features(100)
        .with_min_disk_space_mb(51200)
        .with_min_memory_mb(32768);

    let preflight = Preflight::new()
        .add_check(PreflightCheck::min_samples(1))  // Default overridden by context
        .with_context(ctx);
    }

## Validate or Fail

    #![allow(unused)]
    fn main() {
    // Validate and return error if failed
    let results = preflight.validate(&data)?;

    // This will return Err(PreflightError::ValidationFailed) if checks fail
    }

## Integration with Training

    #![allow(unused)]
    fn main() {
    use entrenar::storage::Preflight;
    use entrenar::train::Trainer;

    fn train_with_preflight(data: &[Vec<f64>]) -> Result<(), Box<dyn std::error::Error>> {
        // Run preflight checks
        let preflight = Preflight::comprehensive();
        preflight.validate(data)?;

        // Proceed with training
        let trainer = Trainer::new(config);
        trainer.fit(data)?;

        Ok(())
    }
    }

## Cargo Run Example

    # Run preflight validation
    cargo run --example preflight_check -- --data train.parquet

    # With verbose output
    cargo run --example preflight_check -- --data train.parquet --verbose

## Sample Report Output

    === Preflight Check Results ===
    Status: FAILED
    Passed: 4, Failed: 1, Warnings: 1, Skipped: 0

    ✓ no_nan_values: No NaN values found
    ✓ no_inf_values: No infinite values found
    ✓ consistent_dimensions: All 10000 rows have 128 features
    ✗ min_samples: Only 500 samples found (minimum: 1000)
    ⚠ no_constant_features: Found 2 constant feature(s): [45, 89]
    ✓ disk_space: 52480 MB available (minimum: 10240 MB)

## Best Practices

1. Run preflight before every training job - Catches issues early
2. Use comprehensive preset for production - Includes environment checks
3. Make class balance optional - Not always applicable
4. Set appropriate thresholds - Too strict causes false positives
5. Log preflight results - Useful for debugging failed runs

## See Also

- MLOps Overview
- Experiment Tracking
- Quality Gates (Jidoka)