# Evaluation Flow Quick Reference

## Entry Points (Current State)

### 1. TaskEvaluator (Recommended for Standard Evaluation)
```rust
use anno::eval::task_evaluator::{TaskEvaluator, TaskEvalConfig};
use anno::eval::task_mapping::Task;

let evaluator = TaskEvaluator::new()?;
let config = TaskEvalConfig {
    tasks: vec![Task::NER],
    datasets: vec![],
    backends: vec!["gliner2".to_string()],
    ..Default::default()
};
let results = evaluator.evaluate_all(config)?;
```
**Use for**: Standard task evaluation on real datasets

### 2. EvalHarness (Legacy/Alternative)
```rust
use anno::eval::harness::{EvalHarness, EvalConfig};

let harness = EvalHarness::new(EvalConfig::default())?;
let results = harness.run_synthetic()?;
```
**Use for**: Synthetic dataset evaluation

### 3. Direct Functions (Low-Level)
```rust
use anno::eval::evaluate_ner_model;

let results = evaluate_ner_model(&model, &test_cases)?;
```
**Use for**: Simple, direct evaluation

### 4. ReportBuilder (Incomplete)
```rust
use anno::eval::report::ReportBuilder;

let report = ReportBuilder::new("model")
    .with_bias_analysis(true)  // Not yet implemented
    .build(&model);
```
**Use for**: Unified reports (when complete)

### 5. Bias Evaluators (Separate)
```rust
use anno::eval::demographic_bias::{DemographicBiasEvaluator, create_diverse_name_dataset};

let evaluator = DemographicBiasEvaluator::new(true);
let names = create_diverse_name_dataset();
let results = evaluator.evaluate_ner(&model, &names);
```
**Use for**: Bias/fairness evaluation

## Data Flow

```
┌─────────────────┐
│   User Code     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐      ┌──────────────────┐
│  TaskEvaluator  │─────▶│  DatasetLoader   │
│  (or Harness)   │      │  (Real datasets) │
└────────┬────────┘      └────────┬─────────┘
         │                        │
         ▼                        ▼
┌─────────────────┐      ┌──────────────────┐
│  BackendFactory │      │  LoadedDataset    │
│  (Create model) │      │  (Gold entities)  │
└────────┬────────┘      └────────┬─────────┘
         │                        │
         └──────────┬─────────────┘
                    ▼
         ┌──────────────────┐
         │  Model Inference  │
         │  (extract_entities)│
         └──────────┬─────────┘
                    ▼
         ┌──────────────────┐
         │  Metrics Compute │
         │  (P/R/F1, etc.)  │
         └──────────┬─────────┘
                    ▼
         ┌──────────────────┐
         │  Results          │
         │  (ComprehensiveEvalResults)│
         └──────────────────┘
```

## Key Abstractions

| Type | Purpose | Location |
|------|---------|----------|
| `Task` | Evaluation task (NER, Coref, etc.) | `task_mapping.rs` |
| `DatasetId` | Real dataset identifier | `loader.rs` |
| `Model` | Backend trait | `lib.rs` |
| `GoldEntity` | Ground truth entity | `datasets.rs` |
| `TaskEvaluator` | Main evaluator | `task_evaluator.rs` |
| `EvalHarness` | Alternative evaluator | `harness.rs` |
| `ComprehensiveEvalResults` | Standard results | `task_evaluator.rs` |
| `BiasEvaluator` | Bias evaluation | `*_bias.rs` |

## Common Patterns

### Pattern 1: Standard Evaluation
```rust
TaskEvaluator::new()?
    .evaluate_all(TaskEvalConfig { ... })
```

### Pattern 2: Bias Evaluation
```rust
DemographicBiasEvaluator::new(true)
    .evaluate_ner(&model, &names)
```

### Pattern 3: Combined
```rust
// Run both separately, combine results
let standard = task_evaluator.evaluate_all(...)?;
let bias = bias_evaluator.evaluate_ner(...)?;
```

## Configuration Complexity

### TaskEvalConfig (11 fields)
- `tasks`, `datasets`, `backends`
- `max_examples`, `seed`
- `require_cached`
- `relation_threshold`
- `robustness`, `compute_familiarity`
- `temporal_stratification`, `confidence_intervals`

### EvalConfig (9 fields)
- `max_examples_per_dataset`
- `breakdown_by_difficulty`, `breakdown_by_domain`, `breakdown_by_type`
- `warmup`, `warmup_iterations`
- `min_confidence`
- `cache_dir`, `normalize_types`

### BiasDatasetConfig (7 fields)
- `frequency_weighted`, `validate_distributions`
- `min_samples_per_category`
- `evaluation_seeds`
- `confidence_level`, `detailed`

## Performance Features

| Feature | Status | Impact |
|---------|--------|--------|
| Parallel processing | ✅ Implemented | 2-4x speedup |
| Progress reporting | ✅ Implemented | UX improvement |
| Async support | ❌ Not implemented | Would help I/O |
| Batch processing | ❌ Not implemented | 1.5-2x potential |
| Session pooling | ⚠️ Available but not used | 1.2-1.5x potential |

## Integration Status

| Component | Integration | Status |
|-----------|-------------|--------|
| Bias evaluation | `ReportBuilder` | ⚠️ Partial (structure exists, not populated) |
| Bias evaluation | `TaskEvaluator` | ❌ Not integrated |
| Calibration | `ReportBuilder` | ❌ Not integrated |
| Data quality | `ReportBuilder` | ❌ Not integrated |
| Standard evaluation | CLI | ✅ Integrated |
| Bias evaluation | CLI | ❌ Not integrated |

## File Sizes (Maintainability Concern)

| File | Lines | Issue |
|------|-------|-------|
| `loader.rs` | 4425 | Too large |
| `abstract_anaphora.rs` | 3519 | Too large |
| `task_evaluator.rs` | 2936 | Too large |
| `coref_metrics.rs` | 2046 | Large |
| `demographic_bias.rs` | 1854 | Large |

## Known Issues

1. **Placeholder Implementations**
   - Runtime capability detection (`task_mapping.rs:332`)
   - Stratified metrics (`task_evaluator.rs:2840`)
   - Per-type metrics computation

2. **Incomplete Integrations**
   - `ReportBuilder.bias` always `None`
   - Bias not in `TaskEvaluator` pipeline
   - Calibration not integrated

3. **Code Duplication**
   - Parallel vs sequential code paths
   - Similar evaluation loops
   - Repeated error handling

4. **String-Based Backend Names**
   - No compile-time checking
   - Must match strings exactly
   - Error-prone

## Recommendations Priority

### High Priority (Immediate Impact)
1. ✅ Unify entry points (create single `EvalSystem`)
2. ✅ Complete `ReportBuilder` integration
3. ✅ Split large files (2000+ lines)

### Medium Priority (1-2 months)
4. Replace string backend names with enum
5. Implement runtime capability detection
6. Add async support

### Low Priority (3-6 months)
7. Unified dataset abstraction
8. Unified metrics framework
9. Performance benchmarking suite

