Benchmark Commands

The entrenar-bench CLI provides tools for distillation benchmarking, hyperparameter sweeps, and cost-performance analysis.

Commands Overview

entrenar-bench <COMMAND>

Commands:
  temperature       Sweep temperature hyperparameter
  alpha             Sweep alpha hyperparameter
  compare           Compare distillation strategies
  ablation          Run ablation study
  cost-performance  Analyze cost vs performance trade-offs
  recommend         Recommend configurations based on constraints

temperature

Run a temperature hyperparameter sweep for knowledge distillation.

entrenar-bench temperature [OPTIONS]

Options

Example

# Default temperature sweep
entrenar-bench temperature

# Custom range with more granularity
entrenar-bench temperature --start 2.0 --end 6.0 --step 0.25

# More runs for statistical significance
entrenar-bench temperature --runs 5 --format json

Output

Temperature Sweep Results
========================

Temp  | Accuracy | Loss   | Std Dev
------|----------|--------|--------
1.0   | 0.823    | 0.412  | ±0.008
1.5   | 0.841    | 0.387  | ±0.006
2.0   | 0.856    | 0.358  | ±0.005
...

Best temperature: 4.0 (accuracy: 0.872)

alpha

Run an alpha (interpolation weight) hyperparameter sweep.

entrenar-bench alpha [OPTIONS]

Options

Example

# Default alpha sweep
entrenar-bench alpha

# Fine-grained sweep around expected optimum
entrenar-bench alpha --start 0.3 --end 0.7 --step 0.05

compare

Compare multiple distillation strategies head-to-head.

entrenar-bench compare [OPTIONS]

Options

Example

# Compare all strategies
entrenar-bench compare --strategies kd,progressive,attention,mse,combined

# Compare specific strategies
entrenar-bench compare --strategies kd,progressive --runs 10

Output

Strategy Comparison Results
===========================

Strategy     | Accuracy | Loss   | Time (s) | Memory (GB)
-------------|----------|--------|----------|------------
kd           | 0.872    | 0.298  | 145.2    | 12.4
progressive  | 0.881    | 0.287  | 312.8    | 14.2
attention    | 0.878    | 0.291  | 198.4    | 16.8
mse          | 0.845    | 0.342  | 98.6     | 10.2
combined     | 0.889    | 0.276  | 425.1    | 18.4

Statistical significance (p < 0.05):
- progressive > kd
- combined > progressive

ablation

Run an ablation study to understand component contributions.

entrenar-bench ablation [OPTIONS]

Options

Example

# Ablation study on distillation components
entrenar-bench ablation \
  --base config.yaml \
  --components "temperature,attention_loss,layer_matching"

Output

Ablation Study Results
======================

Configuration              | Accuracy | Δ Accuracy
---------------------------|----------|----------
Full model                 | 0.889    | baseline
- temperature scaling      | 0.856    | -0.033
- attention loss           | 0.871    | -0.018
- layer matching           | 0.882    | -0.007
- temp - attn              | 0.843    | -0.046

cost-performance

Analyze cost vs performance trade-offs for different configurations.

entrenar-bench cost-performance [OPTIONS]

Options

GPU Cost Models

Example

# Analyze with A100 pricing
entrenar-bench cost-performance --gpu a100-80gb

# Analyze with budget GPU
entrenar-bench cost-performance --gpu t4 --format json

Output

Cost-Performance Analysis (A100-80GB @ $3.00/hr)
================================================

Config           | Hours | Cost   | Accuracy | Pareto
-----------------|-------|--------|----------|-------
LoRA r=8         | 2.1   | $6.30  | 0.845    | Yes
LoRA r=16        | 3.2   | $9.60  | 0.862    | Yes
LoRA r=32        | 5.8   | $17.40 | 0.871    | No
LoRA r=64        | 10.4  | $31.20 | 0.878    | Yes
Full fine-tune   | 48.2  | $144.60| 0.882    | Yes

Pareto Frontier: 4 configurations
Cost efficiency winner: LoRA r=8 (0.134 acc/$)

recommend

Get configuration recommendations based on constraints.

entrenar-bench recommend [OPTIONS]

Options

Example

# Budget-constrained recommendation
entrenar-bench recommend --max-cost 50

# Accuracy-constrained recommendation
entrenar-bench recommend --min-accuracy 0.85

# Multiple constraints
entrenar-bench recommend \
  --max-cost 100 \
  --min-accuracy 0.87 \
  --max-memory 16 \
  --gpu v100

Output

Recommendations (Budget: $50, Min Accuracy: 0.85)
=================================================

Recommended Configuration:
  Method: LoRA
  Rank: 32
  Learning Rate: 1e-4
  Batch Size: 8

Expected Results:
  Accuracy: 0.871
  Training Time: 5.8 hours
  Cost: $17.40
  Memory: 14.2 GB

Rationale:
  - Best accuracy within budget
  - 2.9x cost savings vs full fine-tuning
  - Pareto optimal configuration

Alternative Options:
  1. LoRA r=16: $9.60, 0.862 acc (budget-friendly)
  2. LoRA r=64: $31.20, 0.878 acc (higher accuracy)

Output Formats

All commands support multiple output formats:

Text Format (default)

Human-readable tables and summaries for terminal display.

entrenar-bench temperature --format text

JSON Format

Machine-readable JSON for programmatic processing.

entrenar-bench temperature --format json | jq '.best_temperature'

Example JSON output:

{
  "sweep_type": "temperature",
  "range": {"start": 1.0, "end": 8.0, "step": 0.5},
  "results": [
    {"temperature": 1.0, "accuracy": 0.823, "loss": 0.412, "std_dev": 0.008},
    {"temperature": 1.5, "accuracy": 0.841, "loss": 0.387, "std_dev": 0.006}
  ],
  "best": {"temperature": 4.0, "accuracy": 0.872},
  "statistical_analysis": {
    "mean_accuracy": 0.856,
    "variance": 0.00034
  }
}

Integration with Research Workflow

The benchmark CLI integrates with the research artifact system:

# 1. Initialize research artifact
entrenar research init \
  --id distillation-benchmark \
  --title "Knowledge Distillation Benchmark Study" \
  --type dataset

# 2. Pre-register experiment
entrenar research preregister artifact.yaml \
  --hypothesis "Temperature T=4 is optimal" \
  --methods "Grid search T in [1,8], step 0.5, 5 runs each"

# 3. Run benchmark
entrenar-bench temperature --start 1 --end 8 --step 0.5 --runs 5 \
  --format json > results.json

# 4. Analyze cost-performance
entrenar-bench cost-performance --gpu a100-80gb

# 5. Get recommendations
entrenar-bench recommend --max-cost 100 --min-accuracy 0.85

# 6. Bundle results
entrenar research bundle artifact.yaml --zip

Programmatic API

The benchmark functionality is also available as a Rust library:

#![allow(unused)]
fn main() {
use entrenar_bench::{
    temperature_sweep, compare_strategies,
    CostModel, CostPerformanceAnalysis, Constraints,
};

// Temperature sweep
let result = temperature_sweep(1.0..8.0, 0.5, 3)?;
println!("Best temperature: {}", result.best_param);

// Cost-performance analysis
let cost_model = CostModel::a100_80gb();
let analysis = CostPerformanceAnalysis::new(cost_model);
let pareto = analysis.compute_pareto_frontier(&points);

// Get recommendations
let constraints = Constraints {
    max_cost: Some(50.0),
    min_accuracy: Some(0.85),
    ..Default::default()
};
let recommendations = analysis.recommend(&points, &constraints);
}

See Also

• CLI Overview - General CLI reference
• Research Commands - Research artifact CLI
• Knowledge Distillation - Distillation concepts