# lowess implementation in Rust

[![Crates.io](https://img.shields.io/crates/v/lowess.svg)](https://crates.io/crates/lowess)
[![Documentation](https://docs.rs/lowess/badge.svg)](https://docs.rs/lowess)
[![License](https://img.shields.io/badge/License-AGPL--3.0%20OR%20Commercial-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org)

**High-performance LOWESS (Locally Weighted Scatterplot Smoothing) for Rust** — Production-ready implementation with robust statistics, confidence intervals, and comprehensive features.

## Why This Crate?

- ⚡ **Blazingly Fast**: 4-29× faster performance than Python's statsmodels
- 🎯 **Production-Ready**: Comprehensive error handling, numerical stability, extensive testing
- 📊 **Feature-Rich**: Confidence/prediction intervals, multiple kernels, cross-validation
- 🚀 **Scalable**: Streaming mode and delta optimization
- 🔬 **Scientific**: Validated against R and Python implementations
- 🛠️ **Flexible**: `no_std` support, multiple robustness methods

## NOTE

From release 0.4.0 onwards, the parallelization feature and ndarray support are dropped, in order to make this crate a dependency-free, core LOWESS implementation, ensuring maximum compatibility with other crates or high-level APIs on top of this crate (e.g. `fastLowess`, Python bindings, R bindings, polars plugins, etc.).

👉👉👉 If you need parallelization or ndarray support, please use the `fastLowess` crate at [GitHub](https://github.com/av746/fastLowess) or [Crates.io](https://crates.io/crates/fastLowess). 👈👈👈

## Quick Start

    use lowess::prelude::*;

    let x = vec![1.0, 2.0, 3.0, 4.0, 5.0];
    let y = vec![2.0, 4.1, 5.9, 8.2, 9.8];

    // Basic smoothing
    let result = Lowess::new()
        .fraction(0.5)
        .adapter(Batch)
        .build()?
        .fit(&x, &y)?;

    println!("Smoothed: {:?}", result.y);
    # Ok::<(), LowessError>(())

## Installation

    [dependencies]
    lowess = "0.4"

    # For no_std environments (requires alloc)
    lowess = { version = "0.4", default-features = false }

## Features at a Glance

| Feature                  | Description                             | Use Case                      |
| ------------------------ | --------------------------------------- | ----------------------------- |
| **Robust Smoothing**     | IRLS with Bisquare/Huber/Talwar weights | Outlier-contaminated data     |
| **Confidence Intervals** | Point-wise standard errors & bounds     | Uncertainty quantification    |
| **Cross-Validation**     | Auto-select optimal fraction            | Unknown smoothing parameter   |
| **Multiple Kernels**     | Tricube, Epanechnikov, Gaussian, etc.   | Different smoothness profiles |
| **Streaming Mode**       | Constant memory usage                   | Very large datasets           |
| **Delta Optimization**   | Skip dense regions                      | 10× speedup on dense data     |

## Common Use Cases

### 1. Robust Smoothing (Handle Outliers)

    use lowess::prelude::*;

    # let x = vec![1.0, 2.0, 3.0, 4.0, 5.0];
    # let y = vec![2.0, 4.1, 5.9, 8.2, 9.8];
    let result = Lowess::new()
        .fraction(0.3)
        .iterations(5)                // Robust iterations
        .return_robustness_weights()  // Return outlier weights
        .adapter(Batch)
        .build()?
        .fit(&x, &y)?;

    // Check which points were downweighted
    if let Some(weights) = &result.robustness_weights {
        for (i, &w) in weights.iter().enumerate() {
            if w < 0.1 {
                println!("Point {} is likely an outlier", i);
            }
        }
    }
    # Ok::<(), LowessError>(())

### 2. Uncertainty Quantification

    use lowess::prelude::*;

    # let x = vec![1.0, 2.0, 3.0, 4.0, 5.0];
    # let y = vec![2.0, 4.1, 5.9, 8.2, 9.8];
    let result = Lowess::new()
        .fraction(0.5)
        .weight_function(WeightFunction::Tricube)
        .confidence_intervals(0.95)
        .prediction_intervals(0.95)
        .adapter(Batch)
        .build()?
        .fit(&x, &y)?;

    // Plot confidence bands
    for i in 0..x.len() {
        println!("x={:.1}: y={:.2} CI=[{:.2}, {:.2}]",
            result.x[i],
            result.y[i],
            result.confidence_lower.as_ref().unwrap()[i],
            result.confidence_upper.as_ref().unwrap()[i]
        );
    }
    # Ok::<(), LowessError>(())

### 3. Automatic Parameter Selection

    use lowess::prelude::*;

    # let x = vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0];
    # let y = vec![2.0, 4.1, 5.9, 8.2, 9.8, 12.0, 14.1, 16.0];
    // Let cross-validation find the optimal smoothing fraction
    let result = Lowess::new()
        .cross_validate(&[0.2, 0.3, 0.5, 0.7], CrossValidationStrategy::KFold, Some(5))
        .adapter(Batch)
        .build()?
        .fit(&x, &y)?;

    println!("Optimal fraction: {}", result.fraction_used);
    println!("CV RMSE scores: {:?}", result.cv_scores);
    # Ok::<(), LowessError>(())

### 4. Large Dataset Optimization

    use lowess::prelude::*;

    # let large_x: Vec<f64> = (0..5000).map(|i| i as f64).collect();
    # let large_y: Vec<f64> = large_x.iter().map(|&x| x.sin()).collect();
    // Enable all performance optimizations
    let result = Lowess::new()
        .fraction(0.3)
        .delta(0.01)        // Skip dense regions
        .adapter(Batch)
        .build()?
        .fit(&large_x, &large_y)?;
    # Ok::<(), LowessError>(())

### 5. Production Monitoring

    use lowess::prelude::*;

    # let x = vec![1.0, 2.0, 3.0, 4.0, 5.0];
    # let y = vec![2.0, 4.1, 5.9, 8.2, 9.8];
    let result = Lowess::new()
        .fraction(0.5)
        .iterations(3)
        .return_diagnostics()
        .adapter(Batch)
        .build()?
        .fit(&x, &y)?;

    if let Some(diag) = &result.diagnostics {
        println!("RMSE: {:.4}", diag.rmse);
        println!("R²: {:.4}", diag.r_squared);
        println!("Effective DF: {:.2}", diag.effective_df);

        // Quality checks
        if diag.effective_df < 2.0 {
            eprintln!("Warning: Very low degrees of freedom");
        }
    }
    # Ok::<(), LowessError>(())

### 6. Convenience Constructors

Pre-configured builders for common scenarios:

    use lowess::prelude::*;

    # let x = vec![1.0, 2.0, 3.0, 4.0, 5.0];
    # let y = vec![2.0, 4.1, 5.9, 8.2, 9.8];
    // For noisy data with outliers
    let result = Lowess::<f64>::robust().adapter(Batch).build()?.fit(&x, &y)?;

    // For speed on clean data
    let result = Lowess::<f64>::quick().adapter(Batch).build()?.fit(&x, &y)?;
    # Ok::<(), LowessError>(())

## API Overview

### Builder Methods

    use lowess::prelude::*;

    Lowess::new()
        // Core parameters
        .fraction(0.5)                  // Smoothing span (0, 1], default: 0.67
        .iterations(3)                  // Robustness iterations, default: 3
        .delta(0.01)                    // Interpolation threshold
        
        // Kernel selection
        .weight_function(WeightFunction::Tricube)  // Default
        
        // Robustness method
        .robustness_method(RobustnessMethod::Bisquare)  // Default
        
        // Intervals & diagnostics
        .confidence_intervals(0.95)
        .prediction_intervals(0.95)
        .return_diagnostics()
        .return_residuals()
        .return_robustness_weights()
        
        // Parameter selection
        .cross_validate(&[0.3, 0.5, 0.7], CrossValidationStrategy::KFold, Some(5))
        
        // Convergence
        .auto_converge(1e-4)
        .max_iterations(20)
        
        // Execution mode
        .adapter(Batch)  // or Streaming, Online
        .build()?;       // Build the model

### Result Structure

    pub struct LowessResult<T> {
        pub x: Vec<T>,                          // Sorted x values
        pub y: Vec<T>,                          // Smoothed y values
        pub standard_errors: Option<Vec<T>>,    // Point-wise SE
        pub confidence_lower: Option<Vec<T>>,   // CI lower bound
        pub confidence_upper: Option<Vec<T>>,   // CI upper bound
        pub prediction_lower: Option<Vec<T>>,   // PI lower bound
        pub prediction_upper: Option<Vec<T>>,   // PI upper bound
        pub residuals: Option<Vec<T>>,          // y - fitted
        pub robustness_weights: Option<Vec<T>>, // Final IRLS weights
        pub diagnostics: Option<Diagnostics<T>>,
        pub iterations_used: Option<usize>,     // Actual iterations
        pub fraction_used: T,                   // Selected fraction
        pub cv_scores: Option<Vec<T>>,          // CV RMSE per fraction
    }

## Execution Modes

Choose the right execution mode based on your use case:

### Batch Processing (Standard)

For complete datasets in memory with full feature support:

    use lowess::prelude::*;

    # let x = vec![1.0, 2.0, 3.0, 4.0, 5.0];
    # let y = vec![2.0, 4.1, 5.9, 8.2, 9.8];
    let model = Lowess::new()
        .fraction(0.5)
        .confidence_intervals(0.95)
        .return_diagnostics()
        .adapter(Batch)  // All features supported
        .build()?;

    let result = model.fit(&x, &y)?;
    # Ok::<(), LowessError>(())

### Streaming Processing

For large datasets (>100K points) that don't fit in memory:

    use lowess::prelude::*;

    let mut processor = Lowess::new()
        .fraction(0.3)
        .iterations(2)
        .adapter(Streaming)
        .chunk_size(1000)   // Process 1000 points at a time
        .overlap(100)       // 100 points overlap between chunks
        .build()?;

    // Process data in chunks
    for chunk in data_chunks {
        let result = processor.process_chunk(&chunk.x, &chunk.y)?;
        // Handle results incrementally
    }

    let final_result = processor.finalize()?;
    # Ok::<(), LowessError>(())

### Online/Incremental Processing

For real-time data streams with sliding window:

    use lowess::prelude::*;

    let mut processor = Lowess::new()
        .fraction(0.2)
        .iterations(1)
        .adapter(Online)
        .window_capacity(100)  // Keep last 100 points
        .build()?;

    // Process points as they arrive
    for (x, y) in data_stream {
        if let Some(output) = processor.add_point(x, y)? {
            println!("Smoothed: {}", output.smoothed);
        }
    }
    # Ok::<(), LowessError>(())

## Parameter Selection Guide

### Fraction (Smoothing Span)

- **0.1-0.3**: Local, captures rapid changes (wiggly)
- **0.4-0.6**: Balanced, general-purpose
- **0.7-1.0**: Global, smooth trends only
- **Default: 0.67** (2/3, Cleveland's choice)
- **Use CV** when uncertain

### Robustness Iterations

- **0**: Clean data, speed critical
- **1-2**: Light contamination
- **3**: Default, good balance (recommended)
- **4-5**: Heavy outliers
- **>5**: Diminishing returns

### Kernel Function

- **Tricube** (default): Best all-around, smooth, efficient
- **Epanechnikov**: Theoretically optimal MSE
- **Gaussian**: Very smooth, no compact support
- **Uniform**: Fastest, least smooth (moving average)

### Delta Optimization

- **None**: Small datasets (n < 1000)
- **0.01 × range(x)**: Good starting point for dense data
- **Manual tuning**: Adjust based on data density

## Error Handling

    use lowess::prelude::*;

    # let x = vec![1.0, 2.0, 3.0];
    # let y = vec![2.0, 4.0, 6.0];
    match Lowess::new().adapter(Batch).build()?.fit(&x, &y) {
        Ok(result) => {
            println!("Success: {:?}", result.y);
        },
        Err(LowessError::EmptyInput) => {
            eprintln!("Empty input arrays");
        },
        Err(LowessError::MismatchedInputs { x_len, y_len }) => {
            eprintln!("Length mismatch: x={}, y={}", x_len, y_len);
        },
        Err(LowessError::InvalidFraction(f)) => {
            eprintln!("Invalid fraction: {} (must be in (0, 1])", f);
        },
        Err(e) => {
            eprintln!("Error: {}", e);
        }
    }
    # Ok::<(), LowessError>(())

## Examples

Comprehensive examples are available in the `examples/` directory:

- **`batch_smoothing.rs`** - 8 scenarios covering batch processing
  - Basic smoothing, robust outlier handling, uncertainty quantification
  - Cross-validation, diagnostics, kernel comparisons, robustness methods
  - Quick and robust presets

- **`online_smoothing.rs`** - 6 scenarios for real-time processing
  - Basic streaming, sensor data simulation, outlier handling
  - Window size effects, memory-bounded processing, sliding window behavior

- **`streaming_smoothing.rs`** - 6 scenarios for large datasets
  - Basic chunking, chunk size comparison, overlap strategies
  - Large dataset processing, outlier handling, file-based simulation

Run examples with:

    cargo run --example batch_smoothing
    cargo run --example online_smoothing
    cargo run --example streaming_smoothing

## Feature Flags

- **`std`** (default): Standard library support
- **`dev`**: Exposes internal modules for testing (automatically enabled in development)

### Standard configuration

    [dependencies]
    lowess = "0.4"

### No-std configuration (requires alloc)

    [dependencies]
    lowess = { version = "0.4", default-features = false }

## Validation

This implementation has been extensively validated against:

1. **R's stats::lowess**: Numerical agreement to machine precision
2. **Python's statsmodels**: Validated on 44 test scenarios
3. **Cleveland's original paper**: Reproduces published examples

## MSRV (Minimum Supported Rust Version)

Rust **1.85.0** or later (requires Rust Edition 2024).

## Contributing

Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for:

- Bug reports and feature requests
- Pull request guidelines
- Development workflow
- Testing requirements

## License

MIT License - see [LICENSE](LICENSE) file.

## References

**Original papers:**

- Cleveland, W.S. (1979). "Robust Locally Weighted Regression and Smoothing Scatterplots". _Journal of the American Statistical Association_, 74(368): 829-836. [DOI:10.2307/2286407](https://doi.org/10.2307/2286407)

- Cleveland, W.S. (1981). "LOWESS: A Program for Smoothing Scatterplots by Robust Locally Weighted Regression". _The American Statistician_, 35(1): 54.

**Related implementations:**

- [R stats::lowess](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/lowess.html)
- [Python statsmodels](https://www.statsmodels.org/stable/generated/statsmodels.nonparametric.smoothers_lowess.lowess.html)

## Citation

    @software{lowess_rust_2025,
      author = {Valizadeh, Amir},
      title = {lowess: High-performance LOWESS for Rust},
      year = {2025},
      url = {https://github.com/thisisamirv/lowess},
      version = {0.4.0}
    }

## Author

**Amir Valizadeh**  
📧 <thisisamirv@gmail.com>  
🔗 [GitHub](https://github.com/thisisamirv/lowess)

---

**Keywords**: LOWESS, LOESS, local regression, nonparametric regression, smoothing, robust statistics, time series, bioinformatics, genomics, signal processing
