# Unified Evaluation System

## Overview

The `EvalSystem` provides a **single, consistent entry point** for all evaluation types, replacing the multiple overlapping APIs (TaskEvaluator, EvalHarness, direct functions, etc.).

## Design Goals

- ✅ **Single Entry Point**: One API for all evaluation needs
- ✅ **Unified Results**: Consistent result types across all evaluations
- ✅ **Composable**: Easy to combine standard + bias + calibration
- ✅ **Type-Safe**: Compile-time checking for configurations

## Basic Usage

```rust
use anno::eval::unified_evaluator::EvalSystem;
use anno::StackedNER;

let model = Box::new(StackedNER::default());

let results = EvalSystem::new()
    .with_tasks(vec![Task::NER])
    .with_backends(vec!["stacked".to_string()])
    .with_bias_analysis(true)
    .with_model(model, Some("stacked".to_string()))
    .run()?;

println!("F1: {:.1}%", results.standard.unwrap().f1 * 100.0);
```

## Features

### Standard Evaluation

```rust
EvalSystem::new()
    .with_tasks(vec![Task::NER, Task::IntraDocCoref])
    .with_datasets(vec![DatasetId::WikiGold])
    .with_backends(vec!["gliner2".to_string()])
    .with_max_examples(1000)
    .run()?
```

### Bias Evaluation

```rust
EvalSystem::new()
    .with_bias_analysis(true)
    .with_bias_config(BiasDatasetConfig::default()
        .with_frequency_weighting()
        .with_validation())
    .with_model(model, Some("model_name".to_string()))
    .run()?
```

### Combined Evaluation

```rust
EvalSystem::new()
    // Standard
    .with_tasks(vec![Task::NER])
    .with_backends(vec!["gliner2".to_string()])
    // Bias
    .with_bias_analysis(true)
    // Calibration
    .with_calibration(true)
    // Data quality
    .with_data_quality(true)
    // Model
    .with_model(model, Some("gliner2".to_string()))
    .run()?
```

## Result Structure

```rust
pub struct UnifiedEvalResults {
    pub standard: Option<StandardEvalResults>,      // Standard task evaluation
    pub bias: Option<BiasEvalResults>,              // Bias analysis
    pub calibration: Option<CalibrationEvalResults>, // Calibration metrics
    pub data_quality: Option<DataQualityEvalResults>, // Dataset quality
    pub warnings: Vec<String>,                      // Warnings/notes
    pub metadata: EvalMetadata,                    // Evaluation metadata
}
```

## Migration Guide

### From TaskEvaluator

**Before:**
```rust
let evaluator = TaskEvaluator::new()?;
let config = TaskEvalConfig { ... };
let results = evaluator.evaluate_all(config)?;
```

**After:**
```rust
let results = EvalSystem::new()
    .with_tasks(config.tasks)
    .with_datasets(config.datasets)
    .with_backends(config.backends)
    .run()?;
```

### From EvalHarness

**Before:**
```rust
let harness = EvalHarness::new(EvalConfig::default())?;
let results = harness.run_synthetic()?;
```

**After:**
```rust
let results = EvalSystem::new()
    .with_tasks(vec![Task::NER])
    .run()?;
```

### From Direct Functions

**Before:**
```rust
let results = evaluate_ner_model(&model, &test_cases)?;
```

**After:**
```rust
let results = EvalSystem::new()
    .with_model(Box::new(model), None)
    .run()?;
```

## Benefits

1. **Consistency**: Single API pattern for all evaluation types
2. **Composability**: Easy to combine multiple evaluation types
3. **Type Safety**: Compile-time checking of configurations
4. **Unified Results**: Consistent result structure
5. **Future-Proof**: Easy to add new evaluation types

## Status

- ✅ Core implementation complete
- ✅ Standard evaluation integrated
- ✅ Bias evaluation integrated
- ⚠️ Calibration (placeholder)
- ⚠️ Data quality (placeholder)
- ⚠️ ReportBuilder integration (in progress)

## See Also

- `examples/unified_evaluation.rs` - Complete example
- `docs/EVAL_FLOW_CRITIQUE.md` - Detailed critique
- `docs/EVAL_FLOW_QUICK_REFERENCE.md` - Quick reference

