# TGA Dataset Operation History System - Implementation Guide

## Overview
This document describes the complete implementation of the operation history tracking system for TGADataset.

## Core Data Structures

### 1. OperationRecord
```rust
#[derive(Clone, Debug)]
pub struct OperationRecord {
    pub timestamp: usize,           // Sequential counter
    pub operation_name: String,     // Function name
    pub affected_columns: AffectedColumns,
    pub expr: Option<Expr>,         // Polars expression if applicable
    pub description: String,        // Human-readable description
    pub reversible: bool,           // Can operation be undone?
}
```

### 2. AffectedColumns
```rust
#[derive(Clone, Debug)]
pub enum AffectedColumns {
    Specific(Vec<String>),          // Explicit column names
    All,                             // All columns affected
    Semantic(Vec<ColumnTypes>),     // Semantic column types
}
```

### 3. ColumnHistory
```rust
#[derive(Clone, Debug)]
pub struct ColumnHistory {
    pub column_name: String,
    pub operations: Vec<OperationRecord>,
}

impl ColumnHistory {
    pub fn reversible_count(&self) -> usize;
    pub fn irreversible_count(&self) -> usize;
    pub fn has_irreversible(&self) -> bool;
}
```

## TGADataset Integration

### Updated Structure
```rust
pub struct TGADataset {
    pub frame: LazyFrame,
    pub schema: TGASchema,
    pub oneframeplot: Option<OneFramePlot>,
    pub history_of_operations: Vec<OperationRecord>,  // NEW
}
```

## Core Methods

### 1. Logging Operations
```rust
fn log_operation(
    &mut self,
    name: &str,
    affected: AffectedColumns,
    expr: Option<Expr>,
    description: String,
    reversible: bool,
)
```

**Usage Example:**
```rust
self.log_operation(
    "trim_edges",
    AffectedColumns::All,
    None,
    format!("Trimmed {} rows from left, {} from right", left, right),
    false,  // irreversible - data is removed
);
```

### 2. Query Operations
```rust
// Get all operations affecting a specific column
pub fn operations_on_column(&self, col: &str) -> Vec<&OperationRecord>

// Get complete history for a column
pub fn get_column_history(&self, col: &str) -> ColumnHistory
```

### 3. Display Methods
```rust
// Print all operations
pub fn print_operation_history(&self)

// Print operations for specific column
pub fn print_column_history(&self, col: &str)
```

## Operation Classification

### Reversible Operations (reversible: true)
- **Column transformations**: scale, offset, unit conversions
- **Derived columns**: rates, dimensionless values
- **Algebraic operations**: exp, ln (with new column)
- **Renaming**: column name changes

**Characteristics:**
- Original data preserved
- Can be mathematically inverted
- No data loss

### Irreversible Operations (reversible: false)
- **Row removal**: trim_edges, filter_rows, cut_interval
- **Column deletion**: drop_column
- **Data aggregation**: operations that lose information

**Characteristics:**
- Data permanently removed
- Cannot be undone
- Information loss

## Implementation Examples

### Example 1: Reversible Operation (Unit Conversion)
```rust
pub fn celsius_to_kelvin(mut self) -> Self {
    let col_name = self.schema.temperature.as_ref().unwrap().clone();
    let expr = (col(&col_name) + lit(273.15)).alias(&col_name);
    self.frame = self.frame.with_column(expr.clone());
    
    // Update metadata
    if let Some(meta) = self.schema.columns.get_mut(&col_name) {
        meta.unit = Unit::Kelvin;
        meta.origin = ColumnOrigin::PolarsDerived;
    }

    // Log operation
    self.log_operation(
        "celsius_to_kelvin",
        AffectedColumns::Semantic(vec![ColumnTypes::Temperature]),
        Some(expr),
        format!("Converted {} from Celsius to Kelvin", col_name),
        true,  // reversible: can subtract 273.15
    );

    self
}
```

### Example 2: Irreversible Operation (Trim Edges)
```rust
pub fn trim_edges(mut self, left: usize, right: usize) -> Self {
    let df = self.frame.collect().unwrap();
    let total = df.height();
    let length = total.saturating_sub(left + right);
    let sliced_df = df.slice(left as i64, length);
    let frame = sliced_df.lazy();

    self.log_operation(
        "trim_edges",
        AffectedColumns::All,
        None,  // No Expr - physical row removal
        format!("Trimmed {} rows from left, {} from right", left, right),
        false,  // irreversible: data is permanently removed
    );

    Self {
        frame,
        schema: self.schema,
        oneframeplot: None,
        history_of_operations: self.history_of_operations,
    }
}
```

### Example 3: Derived Column (Rate Calculation)
```rust
pub fn derive_rate(mut self, source_col: &str, new_col: &str) -> Result<Self, TGADomainError> {
    // ... calculation logic ...
    
    let dv = col(source_col).shift(lit(-1)) - col(source_col).shift(lit(1));
    let dt = col(&time).shift(lit(-1)) - col(time).shift(lit(1));
    let rate_expr = (dv.clone() / dt.clone()).alias(new_col);
    
    self.frame = self.frame.with_column(rate_expr.clone());
    
    // ... metadata update ...

    self.log_operation(
        "derive_rate",
        AffectedColumns::Specific(vec![new_col.to_string()]),
        Some(rate_expr),
        format!("Computed rate {} from {} with unit {:?}", new_col, source_col, out_unit),
        true,  // reversible: new column, original preserved
    );

    Ok(self)
}
```

## Usage Patterns

### Pattern 1: Track Processing Pipeline
```rust
let dataset = TGADataset::from_csv("data.csv", "time", "temp", "mass")?
    .trim_edges(5, 5)
    .celsius_to_kelvin()
    .derive_rate("mass", "dm_dt")?
    .dimensionless_mass(0.0, 10.0, "alpha")?;

// View complete history
dataset.print_operation_history();
```

**Output:**
```
=== Operation History ===
[0] trim_edges - Trimmed 5 rows from left, 5 from right (reversible: false)
  Columns: ALL
[1] celsius_to_kelvin - Converted temp from Celsius to Kelvin (reversible: true)
  Semantic: [Temperature]
[2] derive_rate - Computed rate dm_dt from mass with unit MilligramPerSecond (reversible: true)
  Columns: ["dm_dt"]
[3] dimensionless_mass - Computed dimensionless mass alpha from mass (m0=10.5) (reversible: true)
  Columns: ["alpha"]
```

### Pattern 2: Column-Specific History
```rust
// Get history for specific column
dataset.print_column_history("alpha");
```

**Output:**
```
=== History for column 'alpha' ===
Total operations: 2
Reversible: 1
Irreversible: 1
  [0] trim_edges - Trimmed 5 rows from left, 5 from right (reversible: false)
  [3] dimensionless_mass - Computed dimensionless mass alpha from mass (m0=10.5) (reversible: true)
```

### Pattern 3: Programmatic Query
```rust
let history = dataset.get_column_history("mass");

if history.has_irreversible() {
    println!("Warning: Column 'mass' has undergone irreversible operations!");
}

println!("Total transformations: {}", history.operations.len());
println!("Reversible: {}", history.reversible_count());
println!("Irreversible: {}", history.irreversible_count());
```

## Integration Checklist

### For Each Column-Modifying Function:

1. **Identify operation type**:
   - Does it remove data? → `reversible: false`
   - Does it transform data? → `reversible: true`

2. **Determine affected columns**:
   - Specific columns? → `AffectedColumns::Specific(vec![...])`
   - All columns? → `AffectedColumns::All`
   - Semantic type? → `AffectedColumns::Semantic(vec![...])`

3. **Capture expression** (if applicable):
   - Polars operation? → `Some(expr.clone())`
   - Physical operation? → `None`

4. **Add logging call**:
   ```rust
   self.log_operation(
       "function_name",
       affected_columns,
       optional_expr,
       "Human-readable description".to_string(),
       is_reversible,
   );
   ```

## Functions Updated with Logging

### In one_experiment_dataset.rs:
- ✅ `derive_rate0`
- ✅ `derive_rate`
- ✅ `add_numeric_column`

### In exp_kinetics_column_manipulation.rs (see exp_kinetics_column_manipulation_logging.rs):
- ✅ `with_column_expr`
- ✅ `filter_rows`
- ✅ `trim_edges`
- ✅ `trim_null_edges`
- ✅ `rename_column`
- ✅ `drop_column`
- ✅ `celsius_to_kelvin`
- ✅ `seconds_to_hours`
- ✅ `scale_columns`
- ✅ `scale_column`
- ✅ `offset_column`
- ✅ `dimensionless_mass`
- ✅ `conversion`

### Functions to Update (TODO):
- `cut_interval`, `cut_time_interval`, `cut_temperature_interval`, `cut_mass_interval`
- `trim_range`, `trim_range_inverse`
- `trim_null_edges_for_columns`
- `scale_column_in_range_by_reference`, `scale_column_in_its_range`
- `offset_column_in_range_by_reference`, `offset_column_in_its_range`
- `calibrate_mass_from_voltage`, `calibrate_mass`
- `unary_column_op`
- `exp_column`, `ln_column`
- `derive_dimensionless_mass`, `derive_conversion`

## Benefits

1. **Transparency**: Complete audit trail of all transformations
2. **Debugging**: Trace issues back to specific operations
3. **Reproducibility**: Understand exact processing pipeline
4. **Safety**: Identify irreversible operations before they happen
5. **Documentation**: Self-documenting data processing workflow

## Future Enhancements

1. **Undo/Redo**: Implement operation reversal for reversible operations
2. **Export**: Save operation history to JSON/YAML
3. **Replay**: Reconstruct processing pipeline from history
4. **Validation**: Check operation compatibility before execution
5. **Visualization**: Generate flowchart of processing pipeline
