Introduction

Pacha is a unified registry for machine learning artifacts—models, datasets, and training recipes—with full lineage tracking, semantic versioning, and cryptographic integrity.

The Problem

Machine learning projects face critical challenges in artifact management:

1. Model Versioning - How do you track which version of a model is in production?
2. Dataset Provenance - Where did your training data come from? How was it processed?
3. Reproducibility - Can you recreate a model from 6 months ago exactly?
4. Lineage Tracking - Was this model fine-tuned from another? Quantized? Pruned?

The Solution

Pacha provides a content-addressed registry that:

• Deduplicates artifacts using BLAKE3 hashing
• Versions models, datasets, and recipes with semantic versioning
• Documents artifacts with Model Cards and Datasheets
• Tracks lineage through fine-tuning, distillation, quantization, and merging
• Verifies integrity with cryptographic hashes

Core Principles

Toyota Way Methodology

Pacha follows Toyota Way principles:

• Muda (Waste Elimination) - Content-addressed deduplication eliminates duplicate storage
• Jidoka (Built-in Quality) - Cryptographic integrity verification catches corruption
• Kaizen (Continuous Improvement) - Incremental lineage tracking enables iterative improvement

Sovereign AI Stack

Pacha integrates with the Pragmatic AI Labs Sovereign AI Stack:

Quick Start

# Initialize registry
pacha init

# Register a model
pacha model register fraud-detector model.apr -v 1.0.0 -d "Fraud detection model"

# Register a dataset
pacha data register transactions data.ald -v 1.0.0 -p "Transaction data for fraud detection"

# Check model stage
pacha model get fraud-detector -v 1.0.0

# Promote to production
pacha model stage fraud-detector -v 1.0.0 -t production

Architecture Overview

~/.pacha/
├── registry.db      # SQLite metadata
└── objects/         # Content-addressed storage
    ├── ab/
    │   └── cdef1234...
    ├── cd/
    │   └── ef5678...
    └── ...

Key Features

Model Registry

• Semantic versioning (MAJOR.MINOR.PATCH)
• Model Cards (Mitchell et al., 2019)
• Lifecycle stages (development → staging → production → archived)
• Lineage tracking (fine-tuning, distillation, quantization, merging)

Data Registry

• Dataset versioning
• Datasheets (Gebru et al., 2021)
• Provenance tracking (W3C PROV-DM)

Recipe Registry

• Training recipes in TOML
• Hyperparameter specifications
• Environment dependencies
• Hardware requirements

Experiment Tracking

• Run tracking with metrics
• Artifact association
• Git commit tracking
• Duration and status logging

Next Steps

• What is Pacha? - Deep dive into the design philosophy
• Quick Start - Hands-on tutorial
• CLI Reference - Complete command reference

What is Pacha?

MLOps Artifact Management

Semantic Versioning for ML

Reproducibility Principles

System Overview

Content-Addressed Storage

SQLite Metadata Store

BLAKE3 Hashing

Model Versioning

Model Cards

Lifecycle Stages

Model Lineage

Dataset Versioning

Datasheets

Data Provenance

Training Recipes

Hyperparameters

Environment Dependencies

Content Addressing

Deduplication

Integrity Verification

Compression

Lineage Graph

Model Derivation

Fine-Tuning Lineage

Quantization Tracking

Installation

Model Commands

Data Commands

Recipe Commands

Run Commands

Quick Start

This example demonstrates the basic operations of the Pacha registry.

Running the Example

cargo run --example quick_start

What It Does

1. Creates a temporary registry - Uses tempfile for isolated testing
2. Registers a model - Creates a fraud detector with metrics and documentation
3. Registers a dataset - Adds transaction data with a datasheet
4. Queries artifacts - Retrieves and displays model information
5. Transitions stages - Promotes the model from development to staging
6. Shows statistics - Displays registry usage metrics

Code Walkthrough

Creating the Registry

let temp_dir = TempDir::new().expect("Failed to create temp dir");
let config = RegistryConfig::new(temp_dir.path());
let registry = Registry::open(config)?;

Registering a Model

let card = ModelCard::builder()
    .description("Fraud detection model")
    .metrics([("auc", 0.95), ("f1", 0.88)])
    .primary_uses(["Fraud detection in payment transactions"])
    .build();

let model_id = registry.register_model(
    "fraud-detector",
    &ModelVersion::new(1, 0, 0),
    model_data,
    card,
)?;

Registering a Dataset

let datasheet = Datasheet::builder()
    .purpose("Transaction data for fraud detection")
    .creators(["Data Engineering Team"])
    .instance_count(1_000_000)
    .license("Internal Use Only")
    .build();

let dataset_id = registry.register_dataset(
    "transactions",
    &DatasetVersion::new(1, 0, 0),
    dataset_data,
    datasheet,
)?;

Stage Transition

registry.transition_model_stage(
    "fraud-detector",
    &ModelVersion::new(1, 0, 0),
    ModelStage::Staging,
)?;

Expected Output

=== Pacha Quick Start ===

1. Registering a model...
   Registered model ID: 1cd6809b-6b55-48f5-b619-a5ac4930339b

2. Registering a dataset...
   Registered dataset ID: b6cd66ab-5b48-4e7d-ab75-4dd229edc6c8

3. Querying the model...
   Model: fraud-detector:1.0.0
   Stage: development

4. Promoting model to staging...
   New stage: staging

5. Registry statistics:
   Models: 1
   Datasets: 1
   Objects: 2
   Total size: 89 bytes

✅ Quick start complete!

Registering Models

This example demonstrates semantic versioning for ML models.

Running the Example

cargo run --example model_versioning

Semantic Versioning for ML

Pacha uses semantic versioning adapted for machine learning:

Version Examples

Initial Release (1.0.0)

let card = ModelCard::builder()
    .description("Initial fraud detector - logistic regression")
    .metrics([("auc", 0.85)])
    .build();

registry.register_model(
    "fraud-detector",
    &ModelVersion::new(1, 0, 0),
    model_weights,
    card,
)?;

Patch Version (1.0.1) - Quantization

registry.register_model(
    "fraud-detector",
    &ModelVersion::new(1, 0, 1),
    quantized_weights,
    card,
)?;

Minor Version (1.1.0) - Retrained

registry.register_model(
    "fraud-detector",
    &ModelVersion::new(1, 1, 0),
    retrained_weights,
    card,
)?;

Major Version (2.0.0) - New Architecture

registry.register_model(
    "fraud-detector",
    &ModelVersion::new(2, 0, 0),
    transformer_weights,
    card,
)?;

Version Comparison

let v1 = ModelVersion::new(1, 0, 0);
let v1_1 = ModelVersion::new(1, 1, 0);
let v2 = ModelVersion::new(2, 0, 0);

assert!(v1 < v1_1);   // Minor version is higher
assert!(v1_1 < v2);   // Major version is higher

Pre-release Versions

let beta = ModelVersion::new(2, 1, 0).with_prerelease("beta.1");
println!("{}", beta);             // "2.1.0-beta.1"
println!("{}", beta.is_stable()); // false

Tracking Experiments

This example demonstrates experiment tracking in Pacha.

Running the Example

cargo run --example experiment_tracking

Experiment Tracking Features

• Run tracking - Track individual training executions
• Metric logging - Log loss, accuracy, and custom metrics over time
• Hyperparameter storage - Record the exact parameters used
• Run comparison - Find the best run by any metric

Creating a Training Recipe

let recipe = TrainingRecipe::builder()
    .name("fraud-training")
    .version(RecipeVersion::new(1, 0, 0))
    .description("Fraud detection training recipe")
    .hyperparameters(
        Hyperparameters::builder()
            .learning_rate(1e-3)
            .batch_size(32)
            .epochs(10)
            .build(),
    )
    .build();

registry.register_recipe(&recipe)?;

Creating an Experiment Run

let hyperparams = Hyperparameters::builder()
    .learning_rate(1e-4)
    .batch_size(64)
    .epochs(20)
    .build();

let mut run = ExperimentRun::from_recipe(recipe.reference(), hyperparams);
run.start();

Logging Metrics

// During training loop
for epoch in 0..epochs {
    let loss = train_epoch(&model, &data);
    run.log_metric("loss", loss, epoch as u64);
    run.log_metric("accuracy", accuracy, epoch as u64);
}

Completing the Run

run.complete();  // On success
run.fail("Out of memory");  // On failure
run.cancel();  // On cancellation

Finding the Best Run

let runs = registry.list_runs(&recipe.reference())?;

let best = runs
    .iter()
    .filter(|r| r.status == RunStatus::Completed)
    .max_by(|a, b| {
        let auc_a = a.get_metric("auc").unwrap_or(0.0);
        let auc_b = b.get_metric("auc").unwrap_or(0.0);
        auc_a.partial_cmp(&auc_b).unwrap()
    });

Managing Datasets

Training Workflows

Versioning Strategy

Model Documentation

Reproducibility Checklist

CI/CD Integration

Glossary

References

API Reference