Model Lineage

Entrenar provides comprehensive lineage tracking for experiment reproducibility using Lamport timestamps and causal event ordering. The integrity module enables behavioral verification, trace storage policies, and promotion gates for ML model deployment.

Overview

The lineage system consists of three components:

1. LamportTimestamp - Logical clocks for causal ordering across distributed systems
2. CausalLineage - Event tracking with happens-before relationships
3. BehavioralIntegrity - Model promotion gates with metamorphic testing

Lamport Timestamps

Lamport timestamps provide a logical clock for ordering events in distributed systems without relying on synchronized physical clocks.

#![allow(unused)]
fn main() {
use entrenar::integrity::LamportTimestamp;

// Create timestamps for different nodes
let mut node_a = LamportTimestamp::new("node-a");
let mut node_b = LamportTimestamp::new("node-b");

// Increment on local events
node_a.increment();

// Merge when receiving messages (synchronizes clocks)
node_b.merge(&node_a);
node_b.increment();

// Check causal relationships
assert!(node_a.happens_before(&node_b));
}

Happens-Before Relationship

The happens_before() method determines if one event causally precedes another:

#![allow(unused)]
fn main() {
use entrenar::integrity::LamportTimestamp;

let ts1 = LamportTimestamp::with_counter("node-1", 5);
let ts2 = LamportTimestamp::with_counter("node-1", 10);
let ts3 = LamportTimestamp::with_counter("node-2", 7);

// Same node, lower counter = happens before
assert!(ts1.happens_before(&ts2));

// Different nodes may be concurrent
println!("Concurrent: {}", ts1.is_concurrent_with(&ts3));
}

Causal Lineage Tracking

Track experiment events with causal ordering:

#![allow(unused)]
fn main() {
use entrenar::integrity::{CausalLineage, LineageEvent, LineageEventType, LamportTimestamp};

let mut lineage = CausalLineage::new();

// Record experiment lifecycle events
let ts1 = LamportTimestamp::new("trainer-node");
let start = LineageEvent::new(ts1, LineageEventType::RunStarted, "run-001");
lineage.add_event(start);

// Log metrics during training
let ts2 = LamportTimestamp::with_counter("trainer-node", 2);
let metric = LineageEvent::new(ts2, LineageEventType::MetricLogged, "run-001");
lineage.add_event(metric);

// Save artifacts
let ts3 = LamportTimestamp::with_counter("trainer-node", 3);
let artifact = LineageEvent::new(ts3, LineageEventType::ArtifactSaved, "run-001")
    .with_context("checkpoint: epoch_10.pt");
lineage.add_event(artifact);

// Complete the run
let ts4 = LamportTimestamp::with_counter("trainer-node", 4);
let complete = LineageEvent::new(ts4, LineageEventType::RunCompleted, "run-001");
lineage.add_event(complete);
}

Event Types

Querying Events

#![allow(unused)]
fn main() {
use entrenar::integrity::{CausalLineage, LineageEventType};

// Get all events in causal order
let events = lineage.events_in_order();

// Filter by run
let run_events = lineage.events_for_run("run-001");

// Filter by type
let promotions = lineage.events_of_type(LineageEventType::ModelPromoted);

// Get latest event for a run
let latest = lineage.latest_event_for_run("run-001");

// Check if one run precedes another
let precedes = lineage.run_precedes("run-001", "run-002");
}

Trace Storage Policy

Configure how experiment traces are stored and retained:

#![allow(unused)]
fn main() {
use entrenar::integrity::{TraceStoragePolicy, CompressionAlgorithm};

// Custom policy
let policy = TraceStoragePolicy::new(
    CompressionAlgorithm::Zstd,  // Compression algorithm
    30,                           // Retention days
    10 * 1024 * 1024 * 1024,     // Max size (10 GB)
    0.5,                          // Sample rate (50%)
);

// Or use presets
let dev_policy = TraceStoragePolicy::development();
let prod_policy = TraceStoragePolicy::production();
let archive_policy = TraceStoragePolicy::archival();
}

Compression Algorithms

Policy Presets

Sampling

Deterministic sampling ensures consistent trace collection:

#![allow(unused)]
fn main() {
use entrenar::integrity::TraceStoragePolicy;

let policy = TraceStoragePolicy::production();

// Check if a trace should be sampled
if policy.should_sample("trace-12345") {
    // Collect this trace
}

// Same trace ID always returns same result (deterministic)
assert_eq!(
    policy.should_sample("trace-12345"),
    policy.should_sample("trace-12345")
);
}

Behavioral Integrity

Verify model behavior consistency before promotion:

#![allow(unused)]
fn main() {
use entrenar::integrity::{BehavioralIntegrity, BehavioralIntegrityBuilder};

let integrity = BehavioralIntegrityBuilder::new("model-v2.0")
    .equivalence_score(0.98)    // Output consistency
    .syscall_match(0.95)        // System call patterns
    .timing_variance(0.05)      // Inference timing consistency
    .semantic_equiv(0.97)       // Semantic output equivalence
    .test_count(10000)
    .build();

// Check if model passes promotion gate
if integrity.passes_gate(0.9) {
    println!("Model approved for production!");
} else {
    println!("Model failed quality gate");
}
}

Composite Score

The composite score combines multiple metrics with configurable weights:

#![allow(unused)]
fn main() {
use entrenar::integrity::BehavioralIntegrity;

let integrity = BehavioralIntegrity::new(0.9, 0.85, 0.1, 0.88, "model-v1");

// Get composite score (0.0 - 1.0)
let score = integrity.composite_score();
println!("Composite score: {:.1}%", score * 100.0);
}

Metamorphic Violations

Track violations from metamorphic testing:

#![allow(unused)]
fn main() {
use entrenar::integrity::{
    BehavioralIntegrity, MetamorphicViolation, MetamorphicRelationType
};

let mut integrity = BehavioralIntegrity::new(0.9, 0.85, 0.1, 0.88, "model-v1");

// Record a violation
let violation = MetamorphicViolation::new(
    "MV-001",
    MetamorphicRelationType::Identity,
    "Model produces different outputs for identical inputs",
    "Input: [1.0, 2.0, 3.0]",
    "Expected: [0.5, 0.3, 0.2]",
    "Actual: [0.4, 0.4, 0.2]",
    0.7,  // Severity (0.0-1.0)
);

integrity.add_violation(violation);

// Analyze violations
let counts = integrity.violation_counts();
println!("Critical: {}, Warnings: {}, Minor: {}",
    counts.critical, counts.warnings, counts.minor);
}

Metamorphic Relation Types

Assessment Grades

#![allow(unused)]
fn main() {
use entrenar::integrity::{BehavioralIntegrity, IntegrityAssessment};

let integrity = BehavioralIntegrity::new(0.95, 0.92, 0.05, 0.94, "model-v1");

match integrity.assessment() {
    IntegrityAssessment::Excellent => println!("Ready for production"),
    IntegrityAssessment::Good => println!("Minor improvements needed"),
    IntegrityAssessment::Fair => println!("Significant work required"),
    IntegrityAssessment::Poor => println!("Major issues detected"),
    IntegrityAssessment::Critical => println!("Critical violations found"),
}
}

Summary Report

#![allow(unused)]
fn main() {
use entrenar::integrity::BehavioralIntegrity;

let integrity = BehavioralIntegrity::new(0.95, 0.92, 0.05, 0.94, "model-v2.0")
    .with_test_count(10000);

println!("{}", integrity.summary());
// Output:
// Model: model-v2.0
// Composite Score: 94.5%
// Assessment: Excellent
// Violations: 0 critical, 0 warnings, 0 minor
// Tests Run: 10000
// Gate Status: PASS
}

Integration with Experiment Tracking

Combine lineage tracking with behavioral integrity:

#![allow(unused)]
fn main() {
use entrenar::integrity::{
    CausalLineage, LineageEvent, LineageEventType, LamportTimestamp,
    BehavioralIntegrityBuilder,
};

let mut lineage = CausalLineage::new();

// Start training run
let ts1 = LamportTimestamp::new("trainer");
lineage.add_event(LineageEvent::new(ts1, LineageEventType::RunStarted, "run-001"));

// ... training completes ...

// Check behavioral integrity before promotion
let integrity = BehavioralIntegrityBuilder::new("candidate-model")
    .equivalence_score(0.95)
    .syscall_match(0.92)
    .timing_variance(0.08)
    .semantic_equiv(0.94)
    .test_count(5000)
    .build();

if integrity.passes_gate(0.9) {
    // Record promotion with integrity context
    let ts2 = LamportTimestamp::with_counter("trainer", 100);
    let context = format!(
        "score={:.2},assessment={}",
        integrity.composite_score(),
        integrity.assessment()
    );
    let promote = LineageEvent::new(ts2, LineageEventType::ModelPromoted, "run-001")
        .with_context(context);
    lineage.add_event(promote);

    println!("Model promoted to production!");
}
}

Configuration

Configure integrity settings in your training config:

# entrenar.yaml
integrity:
  lineage:
    enabled: true
    node_id: "trainer-001"

  trace_storage:
    compression: zstd
    retention_days: 90
    max_size_gb: 50
    sample_rate: 0.5

  behavioral:
    promotion_threshold: 0.9
    max_timing_variance: 0.2
    require_clean_violations: true