Benchmark Commands - Entrenar - Training & Optimization Library
Description: A comprehensive guide to building neural network training systems with autograd, optimizers, LoRA/QLoRA, and quantization using EXTREME TDD methodology

# Benchmark Commands

The `entrenar-bench` CLI provides tools for distillation benchmarking, hyperparameter sweeps, and cost-performance analysis.

## Commands Overview

    entrenar-bench <COMMAND>

    Commands:
      temperature       Sweep temperature hyperparameter
      alpha             Sweep alpha hyperparameter
      compare           Compare distillation strategies
      ablation          Run ablation study
      cost-performance  Analyze cost vs performance trade-offs
      recommend         Recommend configurations based on constraints

## temperature

Run a temperature hyperparameter sweep for knowledge distillation.

    entrenar-bench temperature [OPTIONS]

### Options

Option | Description
--- | ---
`--start <VALUE>` | Starting temperature (default: 1.0)
`--end <VALUE>` | Ending temperature (default: 8.0)
`--step <VALUE>` | Temperature step size (default: 0.5)
`--runs <N>` | Runs per temperature point (default: 3)
`--format <FORMAT>` | Output format: text, json (default: text)

### Example

    # Default temperature sweep
    entrenar-bench temperature

    # Custom range with more granularity
    entrenar-bench temperature --start 2.0 --end 6.0 --step 0.25

    # More runs for statistical significance
    entrenar-bench temperature --runs 5 --format json

### Output

    Temperature Sweep Results
    ========================

    Temp  | Accuracy | Loss   | Std Dev
    ------|----------|--------|--------
    1.0   | 0.823    | 0.412  | ±0.008
    1.5   | 0.841    | 0.387  | ±0.006
    2.0   | 0.856    | 0.358  | ±0.005
    ...

    Best temperature: 4.0 (accuracy: 0.872)

## alpha

Run an alpha (interpolation weight) hyperparameter sweep.

    entrenar-bench alpha [OPTIONS]

### Options

Option | Description
--- | ---
`--start <VALUE>` | Starting alpha (default: 0.0)
`--end <VALUE>` | Ending alpha (default: 1.0)
`--step <VALUE>` | Alpha step size (default: 0.1)
`--runs <N>` | Runs per alpha point (default: 3)
`--format <FORMAT>` | Output format: text, json (default: text)

### Example

    # Default alpha sweep
    entrenar-bench alpha

    # Fine-grained sweep around expected optimum
    entrenar-bench alpha --start 0.3 --end 0.7 --step 0.05

## compare

Compare multiple distillation strategies head-to-head.

    entrenar-bench compare [OPTIONS]

### Options

Option | Description
--- | ---
`--strategies <LIST>` | Comma-separated strategies: kd, progressive, attention, mse, combined
`--runs <N>` | Runs per strategy (default: 5)
`--format <FORMAT>` | Output format: text, json (default: text)

### Example

    # Compare all strategies
    entrenar-bench compare --strategies kd,progressive,attention,mse,combined

    # Compare specific strategies
    entrenar-bench compare --strategies kd,progressive --runs 10

### Output

    Strategy Comparison Results
    ===========================

    Strategy     | Accuracy | Loss   | Time (s) | Memory (GB)
    -------------|----------|--------|----------|------------
    kd           | 0.872    | 0.298  | 145.2    | 12.4
    progressive  | 0.881    | 0.287  | 312.8    | 14.2
    attention    | 0.878    | 0.291  | 198.4    | 16.8
    mse          | 0.845    | 0.342  | 98.6     | 10.2
    combined     | 0.889    | 0.276  | 425.1    | 18.4

    Statistical significance (p < 0.05):
    - progressive > kd
    - combined > progressive

## ablation

Run an ablation study to understand component contributions.

    entrenar-bench ablation [OPTIONS]

### Options

Option | Description
--- | ---
`--base <CONFIG>` | Base configuration file
`--components <LIST>` | Components to ablate
`--runs <N>` | Runs per configuration (default: 3)
`--format <FORMAT>` | Output format: text, json (default: text)

### Example

    # Ablation study on distillation components
    entrenar-bench ablation \
      --base config.yaml \
      --components "temperature,attention_loss,layer_matching"

### Output

    Ablation Study Results
    ======================

    Configuration              | Accuracy | Δ Accuracy
    ---------------------------|----------|----------
    Full model                 | 0.889    | baseline
    - temperature scaling      | 0.856    | -0.033
    - attention loss           | 0.871    | -0.018
    - layer matching           | 0.882    | -0.007
    - temp - attn              | 0.843    | -0.046

## cost-performance

Analyze cost vs performance trade-offs for different configurations.

    entrenar-bench cost-performance [OPTIONS]

### Options

Option | Description
--- | ---
`--gpu <TYPE>` | GPU type: a100-80gb, v100, t4 (default: a100-80gb)
`--configs <PATH>` | Path to configurations file
`--format <FORMAT>` | Output format: text, json (default: text)

### GPU Cost Models

GPU | Cost/Hour | Memory | Performance Factor
--- | --- | --- | ---
A100-80GB | $3.00 | 80 GB | 1.0x
V100 | $2.00 | 16 GB | 0.6x
T4 | $0.50 | 16 GB | 0.3x

### Example

    # Analyze with A100 pricing
    entrenar-bench cost-performance --gpu a100-80gb

    # Analyze with budget GPU
    entrenar-bench cost-performance --gpu t4 --format json

### Output

    Cost-Performance Analysis (A100-80GB @ $3.00/hr)
    ================================================

    Config           | Hours | Cost   | Accuracy | Pareto
    -----------------|-------|--------|----------|-------
    LoRA r=8         | 2.1   | $6.30  | 0.845    | Yes
    LoRA r=16        | 3.2   | $9.60  | 0.862    | Yes
    LoRA r=32        | 5.8   | $17.40 | 0.871    | No
    LoRA r=64        | 10.4  | $31.20 | 0.878    | Yes
    Full fine-tune   | 48.2  | $144.60| 0.882    | Yes

    Pareto Frontier: 4 configurations
    Cost efficiency winner: LoRA r=8 (0.134 acc/$)

## recommend

Get configuration recommendations based on constraints.

    entrenar-bench recommend [OPTIONS]

### Options

Option | Description
--- | ---
`--max-cost <USD>` | Maximum budget in USD
`--min-accuracy <VALUE>` | Minimum required accuracy (0.0-1.0)
`--max-time <HOURS>` | Maximum training time in hours
`--max-memory <GB>` | Maximum GPU memory in GB
`--gpu <TYPE>` | GPU type for cost calculation
`--format <FORMAT>` | Output format: text, json (default: text)

### Example

    # Budget-constrained recommendation
    entrenar-bench recommend --max-cost 50

    # Accuracy-constrained recommendation
    entrenar-bench recommend --min-accuracy 0.85

    # Multiple constraints
    entrenar-bench recommend \
      --max-cost 100 \
      --min-accuracy 0.87 \
      --max-memory 16 \
      --gpu v100

### Output

    Recommendations (Budget: $50, Min Accuracy: 0.85)
    =================================================

    Recommended Configuration:
      Method: LoRA
      Rank: 32
      Learning Rate: 1e-4
      Batch Size: 8

    Expected Results:
      Accuracy: 0.871
      Training Time: 5.8 hours
      Cost: $17.40
      Memory: 14.2 GB

    Rationale:
      - Best accuracy within budget
      - 2.9x cost savings vs full fine-tuning
      - Pareto optimal configuration

    Alternative Options:
      1. LoRA r=16: $9.60, 0.862 acc (budget-friendly)
      2. LoRA r=64: $31.20, 0.878 acc (higher accuracy)

## Output Formats

All commands support multiple output formats:

### Text Format (default)

Human-readable tables and summaries for terminal display.

    entrenar-bench temperature --format text

### JSON Format

Machine-readable JSON for programmatic processing.

    entrenar-bench temperature --format json | jq '.best_temperature'

Example JSON output:

    {
      "sweep_type": "temperature",
      "range": {"start": 1.0, "end": 8.0, "step": 0.5},
      "results": [
        {"temperature": 1.0, "accuracy": 0.823, "loss": 0.412, "std_dev": 0.008},
        {"temperature": 1.5, "accuracy": 0.841, "loss": 0.387, "std_dev": 0.006}
      ],
      "best": {"temperature": 4.0, "accuracy": 0.872},
      "statistical_analysis": {
        "mean_accuracy": 0.856,
        "variance": 0.00034
      }
    }

## Integration with Research Workflow

The benchmark CLI integrates with the research artifact system:

    # 1. Initialize research artifact
    entrenar research init \
      --id distillation-benchmark \
      --title "Knowledge Distillation Benchmark Study" \
      --type dataset

    # 2. Pre-register experiment
    entrenar research preregister artifact.yaml \
      --hypothesis "Temperature T=4 is optimal" \
      --methods "Grid search T in [1,8], step 0.5, 5 runs each"

    # 3. Run benchmark
    entrenar-bench temperature --start 1 --end 8 --step 0.5 --runs 5 \
      --format json > results.json

    # 4. Analyze cost-performance
    entrenar-bench cost-performance --gpu a100-80gb

    # 5. Get recommendations
    entrenar-bench recommend --max-cost 100 --min-accuracy 0.85

    # 6. Bundle results
    entrenar research bundle artifact.yaml --zip

## Programmatic API

The benchmark functionality is also available as a Rust library:

    #![allow(unused)]
    fn main() {
    use entrenar_bench::{
        temperature_sweep, compare_strategies,
        CostModel, CostPerformanceAnalysis, Constraints,
    };

    // Temperature sweep
    let result = temperature_sweep(1.0..8.0, 0.5, 3)?;
    println!("Best temperature: {}", result.best_param);

    // Cost-performance analysis
    let cost_model = CostModel::a100_80gb();
    let analysis = CostPerformanceAnalysis::new(cost_model);
    let pareto = analysis.compute_pareto_frontier(&points);

    // Get recommendations
    let constraints = Constraints {
        max_cost: Some(50.0),
        min_accuracy: Some(0.85),
        ..Default::default()
    };
    let recommendations = analysis.recommend(&points, &constraints);
    }

## See Also

- CLI Overview - General CLI reference
- Research Commands - Research artifact CLI
- Knowledge Distillation - Distillation concepts