---
title: Performance Tuning Guide
description: Optimize Spring Batch RS jobs for maximum throughput and efficiency
sidebar:
  order: 2
---

import { Card, CardGrid, Aside, Tabs, TabItem } from '@astrojs/starlight/components';

This guide covers performance optimization strategies for Spring Batch RS applications.

## Quick Wins

<CardGrid>
  <Card title="Chunk Size" icon="rocket">
    Increase chunk size to 500-1000 for large datasets
  </Card>
  <Card title="Connection Pooling" icon="star">
    Configure appropriate pool sizes for database operations
  </Card>
  <Card title="Buffering" icon="list">
    Use buffered I/O for file operations
  </Card>
  <Card title="Batch Operations" icon="setting">
    Leverage bulk inserts and batch APIs
  </Card>
</CardGrid>

---

## Chunk Size Optimization

### Understanding Chunk Size

Chunk size determines how many items are processed before writing to the destination.

**Small Chunks (10-50):**
- Lower memory usage
- More frequent I/O operations
- Better for debugging
- More transaction overhead

**Large Chunks (500-1000):**
- Higher memory usage
- Fewer I/O operations
- Better throughput
- Fewer transaction commits

### Recommended Chunk Sizes by Use Case

| Use Case | Recommended Size | Reason |
|----------|-----------------|--------|
| File to File | 500-1000 | Minimize I/O overhead |
| File to Database | 500-1000 | Batch inserts are efficient |
| Database to File | 100-500 | Balance query overhead and memory |
| Database to Database | 500-1000 | Bulk operations on both sides |
| Complex Processing | 100-200 | Manage memory for transformations |
| Testing/Development | 10-50 | Easier debugging |

### Example Configuration

```rust
// Small chunks for development
let step = StepBuilder::new("dev-process")
    .chunk::<Input, Output>(10)  // Small for debugging
    .reader(&reader)
    .writer(&writer)
    .build();

// Large chunks for production
let step = StepBuilder::new("prod-process")
    .chunk::<Input, Output>(1000)  // Large for throughput
    .reader(&reader)
    .writer(&writer)
    .build();
```

---

## Database Performance

### Connection Pool Configuration

<Tabs>
  <TabItem label="PostgreSQL">
    ```rust
    use sqlx::postgres::PgPoolOptions;

    let pool = PgPoolOptions::new()
        .max_connections(10)          // Pool size
        .min_connections(2)           // Minimum idle connections
        .acquire_timeout(Duration::from_secs(30))
        .idle_timeout(Duration::from_secs(600))
        .max_lifetime(Duration::from_secs(1800))
        .connect("postgres://...").await?;
    ```
  </TabItem>
  <TabItem label="MySQL">
    ```rust
    use sqlx::mysql::MySqlPoolOptions;

    let pool = MySqlPoolOptions::new()
        .max_connections(10)
        .min_connections(2)
        .connect("mysql://...").await?;
    ```
  </TabItem>
  <TabItem label="SQLite">
    ```rust
    use sqlx::sqlite::SqlitePoolOptions;

    let pool = SqlitePoolOptions::new()
        .max_connections(5)  // SQLite has lower concurrency
        .connect("sqlite://...").await?;
    ```
  </TabItem>
</Tabs>

**Pool Size Guidelines:**
- Single reader + single writer: 5-10 connections
- Multiple concurrent jobs: 10-20 connections
- High concurrency: 20-50 connections
- Don't exceed database max_connections limit

### Database Reader Optimization

```rust
// Use appropriate page size
let reader = RdbcItemReaderBuilder::<Record>::new()
    .postgres(pool)
    .query("SELECT * FROM large_table ORDER BY id")  // ORDER BY for consistency
    .with_page_size(500)  // Match or exceed chunk size
    .build_postgres();

// For very large datasets, add WHERE clause
let reader = RdbcItemReaderBuilder::<Record>::new()
    .postgres(pool)
    .query("SELECT * FROM large_table WHERE created_at > '2024-01-01' ORDER BY id")
    .with_page_size(1000)
    .build_postgres();
```

### Database Writer Optimization

```rust
// Use bulk inserts with appropriate chunk size
let writer = PostgresItemWriterBuilder::new()
    .pool(pool)
    .table("target_table")
    .binder(|query, records: &Record| {
        // Batch bind all records at once
        query.push_values(records, |mut b, record| {
            b.push_bind(&record.field1)
             .push_bind(&record.field2)
             .push_bind(&record.field3);
        });
    })
    .build();

let step = StepBuilder::new("bulk-insert")
    .chunk::<Record, Record>(1000)  // Large chunks for bulk inserts
    .reader(&reader)
    .writer(&writer)
    .build();
```

### Index Optimization

<Aside type="caution">
**Critical for Performance:**

1. **Add indexes** on columns in WHERE clauses
2. **Add indexes** on ORDER BY columns (for pagination)
3. **Consider composite indexes** for multi-column queries
4. **Monitor index usage** with EXPLAIN ANALYZE
5. **Drop indexes before bulk inserts**, rebuild after
</Aside>

```sql
-- Good: Index on filter and order columns
CREATE INDEX idx_records_created_id ON records(created_at, id);

-- Query uses index effectively
SELECT * FROM records WHERE created_at > '2024-01-01' ORDER BY id;
```

---

## File I/O Performance

### Buffered Reading

```rust
use std::io::BufReader;
use std::fs::File;

// Inefficient: No buffering
let file = File::open("large.csv")?;
let reader = CsvItemReaderBuilder::new()
    .from_reader(file);  // Direct file access

// Efficient: Buffered I/O
let file = File::open("large.csv")?;
let buffered = BufReader::with_capacity(64 * 1024, file);  // 64KB buffer
let reader = CsvItemReaderBuilder::new()
    .from_reader(buffered);
```

### Buffered Writing

```rust
use std::io::BufWriter;

// Efficient: Buffered writing
let file = File::create("output.csv")?;
let buffered = BufWriter::with_capacity(64 * 1024, file);
let writer = CsvItemWriterBuilder::new()
    .from_writer(buffered);
```

### Memory-Mapped Files

For very large files, consider memory-mapped I/O:

```rust
use memmap2::Mmap;
use std::fs::File;

let file = File::open("huge.csv")?;
let mmap = unsafe { Mmap::map(&file)? };
let reader = CsvItemReaderBuilder::new()
    .from_reader(&mmap[..]);
```

---

## Memory Management

### Monitoring Memory Usage

```rust
use sysinfo::{System, SystemExt};

fn log_memory_usage() {
    let mut system = System::new_all();
    system.refresh_all();

    let used = system.used_memory();
    let total = system.total_memory();

    println!("Memory: {:.2} GB / {:.2} GB ({:.1}%)",
        used as f64 / 1_000_000_000.0,
        total as f64 / 1_000_000_000.0,
        (used as f64 / total as f64) * 100.0
    );
}
```

### Memory-Efficient Patterns

**1. Clone Only When Necessary:**
```rust
// Inefficient: Unnecessary clone
impl ItemProcessor<Record, Record> for MyProcessor {
    fn process(&self, item: &Record) -> ItemProcessorResult<Record> {
        let mut cloned = item.clone();  // Expensive
        cloned.field = transform(cloned.field);
        Ok(cloned)
    }
}

// Efficient: Transform in-place when possible
impl ItemProcessor<Record, Record> for MyProcessor {
    fn process(&self, item: &Record) -> ItemProcessorResult<Record> {
        Ok(Record {
            field: transform(&item.field),
            ..item.clone()  // Only clone necessary parts
        })
    }
}
```

**2. Use References for Large Strings:**
```rust
use std::borrow::Cow;

struct Processor;

impl ItemProcessor<String, String> for Processor {
    fn process(&self, item: &String) -> ItemProcessorResult<String> {
        // Use Cow to avoid unnecessary cloning
        let result: Cow<str> = if item.contains("special") {
            Cow::Owned(item.to_uppercase())
        } else {
            Cow::Borrowed(item)
        };

        Ok(result.into_owned())
    }
}
```

**3. Stream Processing:**
```rust
// Process items one at a time without buffering
let step = StepBuilder::new("stream-process")
    .chunk::<Input, Output>(1)  // Process immediately
    .reader(&reader)
    .processor(&processor)
    .writer(&writer)
    .build();
```

---

## Parallel Processing Considerations

### Single-Threaded Model

Spring Batch RS uses a single-threaded model per step execution. For parallel processing:

**Option 1: Multiple Job Instances**
```rust
use std::thread;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let handles: Vec<_> = (0..4).map(|i| {
        thread::spawn(move || {
            process_partition(i)
        })
    }).collect();

    for handle in handles {
        handle.join().unwrap()?;
    }

    Ok(())
}

fn process_partition(partition_id: usize) -> Result<(), Box<dyn std::error::Error>> {
    // Each thread processes a partition
    let reader = CsvItemReaderBuilder::new()
        .from_path(format!("data_part_{}.csv", partition_id))?;

    // ... process partition
    Ok(())
}
```

**Option 2: Async Steps with Tokio**
```rust
use tokio::task;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let tasks = (0..4).map(|i| {
        task::spawn(async move {
            process_partition(i).await
        })
    });

    futures::future::try_join_all(tasks).await?;

    Ok(())
}
```

---

## Monitoring and Profiling

### Basic Metrics

```rust
use std::time::Instant;

let start = Instant::now();

let mut execution = StepExecution::new("process");
step.execute(&mut execution)?;

let duration = start.elapsed();
let items = execution.read_count();
let throughput = items as f64 / duration.as_secs_f64();

println!("Processed {} items in {:?}", items, duration);
println!("Throughput: {:.2} items/sec", throughput);
println!("Write count: {}", execution.write_count());
println!("Skip count: {}", execution.skip_count());
```

### Profiling with Criterion

```toml
[dev-dependencies]
criterion = "0.5"
```

```rust
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn benchmark_processor(c: &mut Criterion) {
    let processor = MyProcessor::new();

    c.bench_function("process item", |b| {
        b.iter(|| {
            let item = black_box(create_test_item());
            processor.process(&item)
        })
    });
}

criterion_group!(benches, benchmark_processor);
criterion_main!(benches);
```

### Flamegraph Profiling

```bash
# Install cargo-flamegraph
cargo install flamegraph

# Run with profiling
cargo flamegraph --bin my-batch-job

# Open flamegraph.svg in browser
```

---

## Real-World Optimization Example

### Before Optimization

```rust
// Slow: Small chunks, no buffering, unnecessary clones
let reader = CsvItemReaderBuilder::new()
    .from_path("data.csv")?;  // No buffering

let processor = SlowProcessor;  // Clones everything

let writer = PostgresItemWriterBuilder::new()
    .pool(small_pool)  // 2 connections only
    .table("target")
    .binder(|query, record| { /* ... */ })
    .build();

let step = StepBuilder::new("slow")
    .chunk::<Record, Record>(10)  // Tiny chunks
    .reader(&reader)
    .processor(&processor)
    .writer(&writer)
    .build();

// Result: 100 items/sec
```

### After Optimization

```rust
use std::io::BufReader;

// Fast: Large chunks, buffered I/O, optimized pool
let file = File::open("data.csv")?;
let buffered = BufReader::with_capacity(64 * 1024, file);
let reader = CsvItemReaderBuilder::new()
    .from_reader(buffered);

let processor = FastProcessor;  // Minimal cloning

let pool = PgPoolOptions::new()
    .max_connections(20)  // Larger pool
    .connect("...").await?;

let writer = PostgresItemWriterBuilder::new()
    .pool(pool)
    .table("target")
    .binder(|query, records| {
        // Bulk insert all records
        query.push_values(records, |mut b, r| {
            b.push_bind(&r.field1)
             .push_bind(&r.field2);
        });
    })
    .build();

let step = StepBuilder::new("fast")
    .chunk::<Record, Record>(1000)  // Large chunks
    .reader(&reader)
    .processor(&processor)
    .writer(&writer)
    .build();

// Result: 10,000 items/sec (100x improvement!)
```

---

## Performance Checklist

<Aside type="tip">
**Before Production:**

- [ ] Chunk size >= 500 for large datasets
- [ ] Database queries have appropriate indexes
- [ ] Connection pool size matches workload
- [ ] File I/O uses buffering
- [ ] Processor minimizes cloning
- [ ] Writer uses bulk operations
- [ ] Memory usage monitored
- [ ] Benchmarks run on production-like data
- [ ] Profiling identifies bottlenecks
- [ ] Error handling doesn't kill performance (use skip_limit)
</Aside>

## Troubleshooting Slow Jobs

### Symptom: Low Throughput

**Check:**
1. Chunk size too small
2. Missing database indexes
3. Network latency (database/FTP)
4. Expensive processor operations
5. Small connection pool

### Symptom: High Memory Usage

**Check:**
1. Chunk size too large
2. Processor accumulating state
3. Memory leaks in custom implementations
4. Large strings not using Cow
5. Unbounded collections

### Symptom: Slow Database Writes

**Check:**
1. Not using bulk inserts
2. Individual INSERT statements
3. Transaction overhead (too frequent commits)
4. Missing indexes on target table
5. Constraints/triggers slowing inserts

## See Also

- [Feature Flags](/spring-batch-rs/reference/features/) - Enable only what you need
- [Error Handling](/spring-batch-rs/error-handling/) - Fault tolerance patterns
- [Examples](/spring-batch-rs/examples/) - Optimized example code
