Core Concepts

This chapter explains the fundamental concepts behind Entrenar's design and how they work together to provide a complete neural network training system.

Architecture Overview

Entrenar is built on four core pillars:

┌─────────────────────────────────────────────────────────┐
│                    Training Loop                        │
│  (User Code: forward pass, loss, backward, optimize)    │
└─────────────────────────────────────────────────────────┘
                           │
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                  ▼
┌───────────────┐  ┌───────────────┐  ┌───────────────┐
│   Autograd    │  │  Optimizers   │  │   LoRA/QLoRA  │
│   Engine      │  │  (SGD, Adam,  │  │  (Parameter-  │
│   (Gradient   │  │   AdamW, LR   │  │   Efficient   │
│   Computation)│  │   Schedulers) │  │   Fine-Tuning)│
└───────────────┘  └───────────────┘  └───────────────┘
        │                  │                  │
        └──────────────────┼──────────────────┘
                           ▼
                   ┌───────────────┐
                   │     Tensor    │
                   │ (Data + Grad) │
                   └───────────────┘

1. Tensors

Tensors are the fundamental data structure in Entrenar, representing multi-dimensional arrays with optional gradient tracking.

Tensor Creation

#![allow(unused)]
fn main() {
use entrenar::Tensor;

// Scalar (0D)
let scalar = Tensor::from_vec(vec![3.14], false);

// Vector (1D)
let vector = Tensor::from_vec(vec![1.0, 2.0, 3.0], true);

// Matrix (2D) - flattened representation
let matrix = Tensor::from_vec(
    vec![1.0, 2.0,
         3.0, 4.0],  // 2x2 matrix
    true
);

// Random initialization
let weights = Tensor::randn(vec![256], true);  // Normal(0, 1)

// Zero initialization
let bias = Tensor::zeros(vec![128], true);
}

Gradient Tracking

#![allow(unused)]
fn main() {
// Trainable parameter
let w = Tensor::from_vec(vec![1.0, 2.0], true);  // requires_grad=true
assert!(w.requires_grad());

// Frozen parameter (e.g., pretrained base weights)
let frozen = Tensor::from_vec(vec![1.0, 2.0], false);  // requires_grad=false
assert!(!frozen.requires_grad());
}

Tensor Operations

#![allow(unused)]
fn main() {
// Arithmetic operations
let a = Tensor::from_vec(vec![1.0, 2.0], true);
let b = Tensor::from_vec(vec![3.0, 4.0], true);

let c = &a + &b;  // Element-wise addition
let d = &a * &b;  // Element-wise multiplication
let e = &a - &b;  // Element-wise subtraction

// Matrix operations
use entrenar::autograd::ops::matmul;

let result = matmul(&a, &b, rows, cols, batch_size);
}

Key Insight: Tensor operations use references (&) to avoid consuming the original tensors, allowing reuse in computational graphs.

2. Automatic Differentiation (Autograd)

Autograd computes gradients automatically using reverse-mode differentiation (backpropagation).

Computational Graph

Entrenar uses a tape-based computational graph:

#![allow(unused)]
fn main() {
let x = Tensor::from_vec(vec![2.0], true);
let y = &x * &x;           // y = x²  (tape records: mul operation)
let z = &y + &x;           // z = x² + x  (tape records: add operation)

backward(&z);              // Compute dz/dx

println!("dz/dx = {}", x.grad()[0]);  // dz/dx = 2x + 1 = 5.0
}

Tape Structure:

Tape:
  1. Op: Mul(x, x) -> y
  2. Op: Add(y, x) -> z

Backward pass (reverse order):
  1. dz/dz = 1.0
  2. dz/dy = 1.0, dz/dx += 1.0
  3. dy/dx = 2x, dz/dx += 2x * dz/dy = 4.0
  Result: dz/dx = 5.0

Supported Operations

Gradient Checking

Entrenar validates all gradients with finite differences:

#![allow(unused)]
fn main() {
#[test]
fn test_gradient_correctness() {
    let x = Tensor::from_vec(vec![1.0, 2.0], true);
    let y = &x * &x;

    backward(&y);

    // Finite difference: f(x+ε) - f(x-ε) / 2ε
    let epsilon = 1e-3;
    let threshold = 0.2;  // 20% relative error tolerance

    check_gradient(&y, &x, epsilon, threshold);  // ✅ Passes
}
}

Zero-tolerance policy: Every operation has gradient checking tests ensuring mathematical correctness.

3. Optimizers

Optimizers update parameters using computed gradients.

Optimizer Interface

All optimizers share a common interface:

#![allow(unused)]
fn main() {
use entrenar::optim::{SGD, Adam, AdamW};

let mut optimizer = Adam::default_params(learning_rate=0.001);

// Training step
backward(&loss);
optimizer.step(&mut [&mut w1, &mut b1, &mut w2, &mut b2]);

// Zero gradients for next iteration
w1.zero_grad();
b1.zero_grad();
// ... etc
}

SGD (Stochastic Gradient Descent)

#![allow(unused)]
fn main() {
use entrenar::optim::SGD;

let mut sgd = SGD::new(
    learning_rate=0.01,
    momentum=0.9,           // Accelerates convergence
);

// Update rule: v = momentum * v + grad
//              param = param - learning_rate * v
sgd.step(&mut params);
}

Use case: Simple optimization, baseline comparisons

Adam (Adaptive Moment Estimation)

#![allow(unused)]
fn main() {
use entrenar::optim::Adam;

let mut adam = Adam::default_params(learning_rate=0.001);

// Adaptive learning rates per parameter
// m = β1*m + (1-β1)*grad           (1st moment)
// v = β2*v + (1-β2)*grad²          (2nd moment)
// param = param - lr * m̂ / (√v̂ + ε)
adam.step(&mut params);
}

Use case: General-purpose, works well out-of-the-box

AdamW (Adam with Decoupled Weight Decay)

#![allow(unused)]
fn main() {
use entrenar::optim::AdamW;

let mut adamw = AdamW::new(
    learning_rate=0.001,
    weight_decay=0.01,      // L2 regularization
    beta1=0.9,
    beta2=0.999,
    epsilon=1e-8,
);

// Decoupled weight decay: param = param * (1 - wd)
adamw.step(&mut params);
}

Use case: Fine-tuning transformers, improved generalization

Learning Rate Schedulers

#![allow(unused)]
fn main() {
use entrenar::optim::schedulers::CosineScheduler;

let scheduler = CosineScheduler::new(
    initial_lr=0.1,
    min_lr=0.001,
    total_steps=1000,
);

for step in 0..1000 {
    let lr = scheduler.get_lr(step);  // Cosine annealing
    optimizer.set_lr(lr);

    // ... training step ...
}
}

4. LoRA (Low-Rank Adaptation)

LoRA enables parameter-efficient fine-tuning by freezing base weights and training low-rank adapters.

Architecture

Original Layer: W ∈ ℝ^(d_out × d_in)

LoRA Layer:
  Base: W ∈ ℝ^(d_out × d_in)     [FROZEN, requires_grad=false]
  Adapters:
    A ∈ ℝ^(rank × d_in)          [TRAINABLE, requires_grad=true]
    B ∈ ℝ^(d_out × rank)         [TRAINABLE, requires_grad=true]

Output: y = Wx + (α/r)(B(Ax))

Usage

#![allow(unused)]
fn main() {
use entrenar::lora::LoRALayer;

// Pretrained base weights (frozen)
let base_weight = Tensor::from_vec(vec![...], false);

// Create LoRA layer
let lora = LoRALayer::new(
    base_weight,
    d_out=256,
    d_in=256,
    rank=16,       // Low-rank bottleneck
    alpha=32.0,    // Scaling factor
);

// Forward pass
let output = lora.forward(&input);

// Only LoRA adapters receive gradients
backward(&loss);  // base_weight.grad() remains zero
}

Parameter Efficiency

Full Fine-Tuning: 7B parameters trainable
LoRA (rank=64):   8M parameters trainable (0.1%)

Memory savings: 99.9% reduction in trainable parameters

Adapter Persistence

#![allow(unused)]
fn main() {
use entrenar::lora::adapter::{save_adapter, load_adapter};

// Save LoRA adapters (32MB file)
save_adapter(&lora, rank=16, alpha=32.0, "adapter.json")?;

// Load adapters (without full model weights)
let loaded_lora = load_adapter("adapter.json", base_weight)?;
}

Use case: Share fine-tuned adapters without distributing 28GB base model weights

5. QLoRA (Quantized LoRA)

QLoRA reduces memory by 75% through 4-bit quantization of frozen base weights.

4-Bit Quantization

#![allow(unused)]
fn main() {
use entrenar::lora::QLoRALayer;

// Base weights quantized to 4-bit (75% memory reduction)
let qlora = QLoRALayer::new(
    base_weight,
    d_out=4096,
    d_in=4096,
    rank=64,
    alpha=128.0,
);

// On-the-fly dequantization during forward pass
let output = qlora.forward(&input);
}

Memory Comparison

Quantization Details

Block-wise quantization (64 elements per block):
  1. Compute scale factor: s = max(|values|) / 7
  2. Quantize: q = round(value / s)  ∈ [-7, 7]
  3. Store: 4-bit signed integers (15 discrete levels)

Dequantization:
  value = q * s  (full precision restored)

Trade-off: Minimal accuracy loss (<1%) for 75% memory reduction

6. EXTREME TDD Quality

Entrenar is built with zero-tolerance for defects using multiple testing strategies:

Unit Tests

#![allow(unused)]
fn main() {
#[test]
fn test_matmul_correctness() {
    let a = Tensor::from_vec(vec![1.0, 2.0, 3.0, 4.0], false);
    let b = Tensor::from_vec(vec![5.0, 6.0, 7.0, 8.0], false);

    let c = matmul(&a, &b, 2, 2, 1);

    assert_eq!(c.data()[0], 19.0);  // 1*5 + 2*7
    assert_eq!(c.data()[1], 43.0);  // 3*5 + 4*7
}
}

Property-Based Tests

#![allow(unused)]
fn main() {
use proptest::prelude::*;

proptest! {
    #[test]
    fn test_adam_converges(lr in 0.05f32..0.5) {
        let optimizer = Adam::default_params(lr);
        assert!(converges_to_minimum(optimizer, 100));
    }
}
}

Gradient Checking

#![allow(unused)]
fn main() {
#[test]
fn test_relu_gradient() {
    let x = Tensor::from_vec(vec![-1.0, 0.0, 1.0], true);
    let y = relu(&x);

    backward(&y);

    // Finite difference validation (ε=1e-3, threshold=0.2)
    check_gradient(&y, &x, 1e-3, 0.2);
}
}

Mutation Testing

cargo mutants --file src/autograd/ops.rs

# Ensures tests catch intentional bugs
# Target: >80% mutation kill rate

Putting It All Together

Complete Training Workflow

#![allow(unused)]
fn main() {
use entrenar::{Tensor, backward, optim::AdamW, lora::QLoRALayer};

// 1. Load pretrained base weights
let base_weight = load_pretrained_weights("llama-7b.bin");

// 2. Create QLoRA layer (75% memory reduction)
let qlora = QLoRALayer::new(base_weight, 4096, 4096, rank=64, alpha=128.0);

// 3. Initialize optimizer
let mut optimizer = AdamW::new(lr=0.0001, weight_decay=0.01, ...);

// 4. Training loop
for (input, target) in dataloader {
    // Forward pass
    let output = qlora.forward(&input);
    let loss = cross_entropy_loss(&output, &target);

    // Backward pass (only LoRA adapters get gradients)
    backward(&loss);

    // Update (only 8M parameters instead of 7B)
    optimizer.step(&mut qlora.trainable_parameters());

    // Zero gradients
    qlora.zero_grad();
}

// 5. Save adapters (32MB file)
save_adapter(&qlora, "custom_adapter.json")?;
}

Result: Fine-tune 7B parameter model on consumer GPU with 8GB VRAM

Key Takeaways

1. Tensors store data and gradients, enabling automatic differentiation
2. Autograd computes gradients via reverse-mode differentiation on a tape-based graph
3. Optimizers update parameters using various strategies (SGD, Adam, AdamW)
4. LoRA trains low-rank adapters instead of full weights (99.9% parameter reduction)
5. QLoRA quantizes base weights to 4-bit for 75% memory savings
6. EXTREME TDD ensures zero defects through comprehensive testing

What's Next?

• Autograd Engine - Deep dive into automatic differentiation
• Optimizers - Explore optimizer algorithms and theory
• LoRA/QLoRA - Master parameter-efficient fine-tuning
• Examples - See practical applications

Ready to explore the autograd engine? Continue to What is Automatic Differentiation? →