# Bias Evaluation Integration with Evaluation & Dataset System

This document explains how bias evaluations integrate with the main evaluation and dataset infrastructure.

## Architecture Overview

The `anno` evaluation system has two complementary evaluation paths:

### 1. **Standard Task Evaluation** (via `TaskEvaluator`)
- **Purpose**: Evaluate models on real-world datasets for standard tasks (NER, Coref, etc.)
- **Datasets**: Real datasets loaded via `DatasetLoader` (WikiGold, CoNLL, GAP, etc.)
- **Flow**: `Task` → `DatasetId` → `DatasetLoader` → `Backend` → Metrics
- **Example**: Evaluate NER on WikiGold using GLiNER

### 2. **Bias Evaluation** (via bias evaluators)
- **Purpose**: Evaluate fairness/bias across demographic dimensions
- **Datasets**: Synthetic datasets created programmatically (not via `DatasetLoader`)
- **Flow**: `BiasEvaluator` → Synthetic Dataset → `Model` → Bias Metrics
- **Example**: Evaluate demographic bias using `DemographicBiasEvaluator`

## Key Differences

| Aspect | Standard Evaluation | Bias Evaluation |
|--------|-------------------|-----------------|
| **Datasets** | Real datasets (WikiGold, CoNLL, etc.) | Synthetic datasets (programmatically generated) |
| **Dataset Source** | `DatasetLoader` (downloads/caches) | `create_*_dataset()` functions |
| **Task Mapping** | Via `TaskMapping` (task → datasets → backends) | Direct evaluator → model |
| **Purpose** | Measure accuracy/performance | Measure fairness/bias |
| **Integration** | `TaskEvaluator.evaluate_all()` | Standalone evaluators |

## Dataset Types

### Standard Datasets (via `DatasetLoader`)

```rust
use anno::eval::loader::{DatasetLoader, DatasetId};

let loader = DatasetLoader::new()?;
let dataset = loader.load(DatasetId::WikiGold)?;
// Real dataset with gold annotations
```

**Examples**: WikiGold, CoNLL-2003, GAP, PreCo, etc.

### Bias Datasets (synthetic, programmatic)

```rust
use anno::eval::demographic_bias::create_diverse_name_dataset;
use anno::eval::gender_bias::create_winobias_templates;
use anno::eval::temporal_bias::create_temporal_name_dataset;
use anno::eval::length_bias::create_length_varied_dataset;

let names = create_diverse_name_dataset();        // 500+ names
let templates = create_winobias_templates();       // 150 examples
let temporal = create_temporal_name_dataset();     // 200+ names
let length = create_length_varied_dataset();       // 100+ examples
```

**Characteristics**:
- Created programmatically (not downloaded)
- Designed for bias testing (demographic diversity, stereotypes, etc.)
- Include metadata (ethnicity, gender, frequency, etc.)
- Expanded to 5x larger than before

## Integration Patterns

### Pattern 1: Standalone Bias Evaluation

Run bias evaluations independently of standard evaluations:

```rust
use anno::eval::bias_config::BiasDatasetConfig;
use anno::eval::demographic_bias::{DemographicBiasEvaluator, create_diverse_name_dataset};
use anno::RegexNER;

// Create config
let config = BiasDatasetConfig::default()
    .with_frequency_weighting()
    .with_validation();

// Create evaluator and dataset
let evaluator = DemographicBiasEvaluator::with_config(true, config);
let names = create_diverse_name_dataset();

// Evaluate
let results = evaluator.evaluate_ner(&model, &names);

// Analyze results
println!("Parity gap: {:.1}%", results.ethnicity_parity_gap * 100.0);
```

### Pattern 2: Combined Evaluation Workflow

Run both standard and bias evaluations in the same workflow:

```rust
use anno::eval::task_evaluator::{TaskEvaluator, TaskEvalConfig};
use anno::eval::demographic_bias::{DemographicBiasEvaluator, create_diverse_name_dataset};
use anno::eval::task_mapping::Task;

// 1. Standard evaluation on real datasets
let task_evaluator = TaskEvaluator::new()?;
let config = TaskEvalConfig {
    tasks: vec![Task::NER],
    datasets: vec![],  // Use all suitable datasets
    backends: vec!["gliner2".to_string()],
    ..Default::default()
};
let standard_results = task_evaluator.evaluate_all(config)?;

// 2. Bias evaluation on synthetic datasets
let bias_evaluator = DemographicBiasEvaluator::new(true);
let names = create_diverse_name_dataset();
let bias_results = bias_evaluator.evaluate_ner(&model, &names);

// 3. Compare/combine results
println!("Standard F1: {:.1}%", standard_results.results[0].metrics["f1"] * 100.0);
println!("Bias parity gap: {:.1}%", bias_results.ethnicity_parity_gap * 100.0);
```

### Pattern 3: Integrated into Report System

Use `ReportBuilder` to include bias analysis:

```rust
use anno::eval::report::{ReportBuilder, EvalReport};
use anno::RegexNER;

let model = RegexNER::new();
let report = ReportBuilder::new("MyModel")
    .with_core_metrics(true)
    .with_bias_analysis(true)  // Enable bias analysis
    .with_error_analysis(true)
    .build(&model);

// Bias results available in report.bias
if let Some(bias) = &report.bias {
    println!("Bias detected: {}", bias.bias_detected);
    if let Some(demo) = &bias.demographic {
        println!("Max gap: {:.1}%", demo.max_gap * 100.0);
    }
}
```

## Why Separate Systems?

### 1. **Different Data Sources**

**Standard evaluations** need:
- Real-world datasets with gold annotations
- Downloaded/cached from external sources
- Domain-specific (news, biomedical, etc.)

**Bias evaluations** need:
- Controlled synthetic datasets
- Known demographic distributions
- Designed for fairness testing

### 2. **Different Evaluation Goals**

**Standard evaluations** measure:
- Accuracy (precision, recall, F1)
- Performance across entity types
- Robustness to perturbations

**Bias evaluations** measure:
- Fairness (parity gaps, recognition rates)
- Performance across demographic groups
- Intersectional analysis

### 3. **Different Integration Points**

**Standard evaluations**:
- Integrated into `TaskEvaluator` pipeline
- Use `TaskMapping` for task-dataset-backend matching
- Generate `ComprehensiveEvalResults`

**Bias evaluations**:
- Standalone evaluators
- Direct model → evaluator → results
- Generate `*BiasResults` structs

## Task Mapping Integration

Bias evaluations are **not** part of the `Task` enum or `TaskMapping` because:

1. **They're not tasks** - They're analysis methods
2. **They don't use standard datasets** - They use synthetic datasets
3. **They're supplementary** - They complement standard evaluations

However, they can be associated with tasks:

- **Gender Bias** → `Task::IntraDocCoref` (coreference resolution)
- **Demographic Bias** → `Task::NER` (named entity recognition)
- **Temporal Bias** → `Task::NER` (named entity recognition)
- **Length Bias** → `Task::NER` (named entity recognition)

## Dataset Loader Integration

Bias datasets are **not** loaded via `DatasetLoader` because:

1. **They're synthetic** - Created programmatically, not downloaded
2. **They're small** - No need for caching/downloading
3. **They're versioned in code** - Changes tracked in git, not external URLs

However, they follow similar patterns:

```rust
// Standard dataset
let loader = DatasetLoader::new()?;
let dataset = loader.load(DatasetId::WikiGold)?;

// Bias dataset (similar pattern, different source)
let names = create_diverse_name_dataset();  // Programmatic, not loaded
```

## Comprehensive Evaluation Workflow

Here's how to run a complete evaluation including bias:

```rust
use anno::eval::task_evaluator::{TaskEvaluator, TaskEvalConfig};
use anno::eval::task_mapping::Task;
use anno::eval::bias_config::BiasDatasetConfig;
use anno::eval::demographic_bias::{DemographicBiasEvaluator, create_diverse_name_dataset};
use anno::eval::gender_bias::{GenderBiasEvaluator, create_winobias_templates};
use anno::eval::coref_resolver::SimpleCorefResolver;

// 1. Standard NER evaluation
let task_eval = TaskEvaluator::new()?;
let config = TaskEvalConfig {
    tasks: vec![Task::NER],
    datasets: vec![],  // All suitable datasets
    backends: vec!["gliner2".to_string()],
    ..Default::default()
};
let standard_results = task_eval.evaluate_all(config)?;

// 2. Demographic bias evaluation
let bias_config = BiasDatasetConfig::default()
    .with_frequency_weighting()
    .with_validation();
let demo_evaluator = DemographicBiasEvaluator::with_config(true, bias_config);
let names = create_diverse_name_dataset();
let demo_results = demo_evaluator.evaluate_ner(&model, &names);

// 3. Gender bias evaluation (if coreference resolver available)
let resolver = SimpleCorefResolver::default();
let templates = create_winobias_templates();
let gender_evaluator = GenderBiasEvaluator::new(true);
let gender_results = gender_evaluator.evaluate_resolver(&resolver, &templates);

// 4. Combine results
println!("=== Standard Evaluation ===");
println!("F1: {:.1}%", standard_results.results[0].metrics["f1"] * 100.0);

println!("\n=== Bias Evaluation ===");
println!("Demographic parity gap: {:.1}%", demo_results.ethnicity_parity_gap * 100.0);
println!("Gender bias gap: {:.1}%", gender_results.bias_gap * 100.0);
```

## Future Integration Possibilities

### Option 1: Add Bias as a Task Type

```rust
// Could add to Task enum:
Task::BiasAnalysis,  // New task type

// Could add to task mapping:
pub fn task_datasets(task: Task) -> &'static [DatasetId] {
    match task {
        Task::BiasAnalysis => &[],  // Uses synthetic datasets
        // ...
    }
}
```

**Pros**: Unified evaluation pipeline  
**Cons**: Doesn't fit task-dataset-backend model (no real datasets)

### Option 2: Add Bias to TaskEvalConfig

```rust
pub struct TaskEvalConfig {
    // ... existing fields ...
    /// Whether to run bias evaluations alongside standard evaluations
    pub include_bias: bool,
    /// Bias evaluation configuration
    pub bias_config: Option<BiasDatasetConfig>,
}
```

**Pros**: Easy to enable/disable  
**Cons**: Still separate evaluation path

### Option 3: Keep Separate (Current Approach)

**Pros**: 
- Clear separation of concerns
- Flexible (can run independently)
- No coupling between systems

**Cons**:
- Requires manual integration
- Two separate workflows

## Best Practices

1. **Run both evaluations**: Standard + bias for comprehensive assessment
2. **Use same model**: Evaluate same model instance on both
3. **Compare results**: Standard F1 vs. bias parity gaps
4. **Document findings**: Include both in reports
5. **Automate workflow**: Create scripts that run both

## Example: Complete Evaluation Script

```rust
fn comprehensive_evaluation(model: &dyn Model) -> Result<()> {
    // Standard evaluation
    let task_eval = TaskEvaluator::new()?;
    let standard_results = task_eval.evaluate_all(TaskEvalConfig {
        tasks: vec![Task::NER],
        ..Default::default()
    })?;

    // Bias evaluation
    let bias_config = BiasDatasetConfig::default()
        .with_frequency_weighting()
        .with_validation();
    
    let demo_evaluator = DemographicBiasEvaluator::with_config(true, bias_config);
    let names = create_diverse_name_dataset();
    let bias_results = demo_evaluator.evaluate_ner(model, &names);

    // Generate combined report
    println!("=== Evaluation Report ===");
    println!("\nStandard Performance:");
    for result in &standard_results.results {
        println!("  {} on {}: F1={:.1}%",
            result.backend, result.dataset,
            result.metrics["f1"] * 100.0);
    }

    println!("\nBias Analysis:");
    println!("  Demographic parity gap: {:.1}%",
        bias_results.ethnicity_parity_gap * 100.0);
    println!("  Script bias gap: {:.1}%",
        bias_results.script_bias_gap * 100.0);

    if let Some(validation) = &bias_results.distribution_validation {
        println!("  Distribution valid: {}", validation.is_valid);
    }

    Ok(())
}
```

## Summary

- **Bias evaluations are separate** from standard task evaluations
- **They use synthetic datasets** (not loaded via `DatasetLoader`)
- **They're supplementary analyses** that complement standard evaluations
- **They can be integrated** into workflows alongside standard evaluations
- **They follow similar patterns** but serve different purposes

The separation allows both systems to evolve independently while still being used together for comprehensive model assessment.

