   Compiling entrenar v0.2.2 (/home/noah/src/entrenar)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.50s
     Running `target/debug/examples/distillation`
=== Knowledge Distillation Examples ===

1. STANDARD DISTILLATION

Standard temperature-scaled distillation with KL divergence

Configuration:
  Temperature: 3
  Alpha (soft weight): 0.7
  Hard weight: 0.3

Results:
  Teacher logits: [[10, 2, 1], [1, 12, 2], [2, 1, 11]]
  Student logits: [[7, 3, 2], [2, 8, 3], [3, 2, 7]]
  Ground truth labels: [0, 1, 2]
  Distillation loss: 0.8538

Temperature effect:
  T= 1.0 → loss=0.0187
  T= 2.0 → loss=0.3158
  T= 5.0 → loss=1.5673
  T=10.0 → loss=1.9169

Alpha (soft weight) effect:
  α=0.0 → loss=0.0196
  α=0.3 → loss=0.3771
  α=0.5 → loss=0.6154
  α=0.7 → loss=0.8538
  α=1.0 → loss=1.2113

2. MULTI-TEACHER ENSEMBLE DISTILLATION

Distilling knowledge from multiple teachers

Three specialized teachers:
  Teacher 1: Strong on class 0
  Teacher 2: Strong on class 1
  Teacher 3: Strong on class 2

--- Uniform Ensemble ---
Weights: [0.33, 0.33, 0.33]
Ensemble logits (first sample):
  [5.00, 5.33, 4.67]

--- Weighted Ensemble ---
Weights: [0.60, 0.20, 0.20]
Ensemble logits (first sample):
  [7.00, 4.40, 3.60]
  → Biased toward class 0 due to higher teacher 1 weight

Student distillation loss: 0.4878

3. PROGRESSIVE LAYER-WISE DISTILLATION

Layer-wise distillation of intermediate representations

Configuration:
  Number of layers: 3
  Hidden dimension: 3
  Batch size: 2

--- Uniform Layer Weights ---
Weights: [0.33, 0.33, 0.33]
  MSE loss: 0.1000
  Cosine loss: 0.0001

--- Progressive Layer Weights ---
Weights: [0.14, 0.29, 0.57] (later layers weighted more)
  MSE loss: 0.1557
  Cosine loss: 0.0001
  → Higher loss due to emphasis on layer 3 (which has larger error)

--- Combined Distillation ---
Alpha (soft target weight): 0.7
Beta (hidden state weight): 0.3
Combined loss: 0.1186
  = 70% logit distillation + 30% hidden state matching

=== Examples Complete ===

