---
title: Architecture
description: Deep dive into Spring Batch RS architecture and design patterns
sidebar:
  order: 1
---

import { Card, CardGrid, Tabs, TabItem, Aside } from '@astrojs/starlight/components';

# Spring Batch RS Architecture

Spring Batch RS is built on proven batch processing patterns from the Java Spring Batch framework, adapted for Rust's unique strengths in performance and safety.

## High-Level Architecture

```mermaid
graph TB
    subgraph "Application Layer"
        App[Your Application]
    end

    subgraph "Spring Batch RS Core"
        Job[Job]
        JobExec[JobExecution]
        Step[Step]
        StepExec[StepExecution]
    end

    subgraph "Processing Layer"
        ChunkOrient[Chunk-Oriented Processing]
        TaskletProc[Tasklet Processing]
    end

    subgraph "I/O Layer"
        Reader[ItemReader]
        Processor[ItemProcessor]
        Writer[ItemWriter]
        Tasklet[Tasklet]
    end

    subgraph "Data Sources"
        Files[Files<br/>CSV, JSON, XML]
        DB[Databases<br/>PostgreSQL, MySQL, SQLite]
        NoSQL[NoSQL<br/>MongoDB]
        Network[Network<br/>FTP, FTPS]
    end

    App --> Job
    Job --> JobExec
    JobExec --> Step
    Step --> StepExec
    StepExec --> ChunkOrient
    StepExec --> TaskletProc

    ChunkOrient --> Reader
    ChunkOrient --> Processor
    ChunkOrient --> Writer
    TaskletProc --> Tasklet

    Reader -.-> Files
    Reader -.-> DB
    Reader -.-> NoSQL
    Writer -.-> Files
    Writer -.-> DB
    Writer -.-> NoSQL
    Tasklet -.-> Network
    Tasklet -.-> Files

    style Job fill:#3b82f6,stroke:#1e40af,color:#fff
    style Step fill:#10b981,stroke:#059669,color:#fff
    style ChunkOrient fill:#f59e0b,stroke:#d97706,color:#fff
    style TaskletProc fill:#f59e0b,stroke:#d97706,color:#fff
```

## Core Components

### Job

A **Job** represents the entire batch process. It's the top-level container that orchestrates one or more steps.

```rust
use spring_batch_rs::core::job::JobBuilder;

let job = JobBuilder::new()
    .start(&step1)
    .next(&step2)
    .next(&step3)
    .build();

let result = job.run()?;
```

**Key Characteristics:**
- Immutable once created
- Can have multiple steps executed sequentially
- Maintains execution state and metadata
- Provides rollback capabilities on failure

### Step

A **Step** is an independent, sequential phase of a Job. Each step can either process data in chunks or execute a single task.

```mermaid
graph LR
    Step[Step] --> Type{Step Type?}
    Type -->|Chunk-Oriented| Chunk[Read → Process → Write]
    Type -->|Tasklet| Task[Single Task Execution]

    style Step fill:#10b981,color:#fff
    style Chunk fill:#3b82f6,color:#fff
    style Task fill:#f59e0b,color:#fff
```

### Chunk-Oriented Processing

The read-process-write pattern for handling large datasets efficiently.

```mermaid
sequenceDiagram
    participant Step
    participant Reader
    participant Processor
    participant Writer

    loop For each chunk
        Step->>Reader: read(chunk_size)
        Reader-->>Step: items[1..N]

        loop For each item
            Step->>Processor: process(item)
            Processor-->>Step: transformed_item
        end

        Step->>Writer: write(chunk)
        Writer-->>Step: success
    end
```

**Architecture Benefits:**
- **Memory Efficient**: Only loads chunk_size items at a time
- **Transactional**: Commits per chunk, not per item
- **Fault Tolerant**: Can skip failed items within limits
- **Performant**: Batches I/O operations

### ItemReader

Abstracts data retrieval from various sources.

```rust
pub trait ItemReader<T> {
    fn read(&mut self) -> ItemReaderResult<T>;
}
```

**Design Pattern**: Iterator-like pattern with error handling

<CardGrid>
  <Card title="File Readers" icon="document">
    - CsvItemReader
    - JsonItemReader
    - XmlItemReader
  </Card>
  <Card title="Database Readers" icon="seti:db">
    - RdbcItemReader (SQL)
    - OrmItemReader (SeaORM)
    - MongoItemReader
  </Card>
  <Card title="Utility Readers" icon="star">
    - FakeItemReader
    - Custom implementations
  </Card>
</CardGrid>

### ItemProcessor

Transforms and validates items during processing.

```rust
pub trait ItemProcessor<I, O> {
    fn process(&self, item: I) -> ItemProcessorResult<O>;
}
```

**Key Features:**
- Type transformation: `I` → `O`
- Filtering: Return `None` to skip items
- Validation: Return `Err` for invalid items
- Stateless design for parallelization

### ItemWriter

Outputs processed items to destinations.

```rust
pub trait ItemWriter<T> {
    fn write(&mut self, items: &[T]) -> ItemWriterResult;
}
```

**Batch Writing**: Receives chunks of items for efficient I/O

### Tasklet

Single-task operations that don't fit the chunk pattern.

```rust
pub trait Tasklet {
    fn execute(&self, step_execution: &StepExecution)
        -> Result<RepeatStatus, BatchError>;
}
```

**Common Use Cases:**
- File compression (ZIP)
- File transfer (FTP/FTPS)
- Database maintenance
- Cleanup operations
- API calls

## Execution Flow

### Complete Job Execution

```mermaid
stateDiagram-v2
    [*] --> JobStarting
    JobStarting --> StepExecution

    state StepExecution {
        [*] --> ReadChunk
        ReadChunk --> ProcessItems
        ProcessItems --> WriteChunk
        WriteChunk --> MoreData?
        MoreData? --> ReadChunk: Yes
        MoreData? --> [*]: No
    }

    StepExecution --> NextStep?
    NextStep? --> StepExecution: More Steps
    NextStep? --> JobComplete: Done
    JobComplete --> [*]
```

### Error Handling Flow

```mermaid
graph TB
    Read[Read Item] --> Process[Process Item]
    Process --> Success{Success?}
    Success -->|Yes| Write[Write to Chunk]
    Success -->|No| SkipCheck{Skip Limit<br/>Reached?}
    SkipCheck -->|No| Skip[Skip Item & Continue]
    SkipCheck -->|Yes| Fail[Fail Job]
    Write --> MoreItems{More Items?}
    MoreItems -->|Yes| Read
    MoreItems -->|No| Commit[Commit Chunk]
    Skip --> MoreItems

    style Success fill:#10b981,color:#fff
    style SkipCheck fill:#f59e0b,color:#fff
    style Fail fill:#ef4444,color:#fff
    style Commit fill:#3b82f6,color:#fff
```

## Design Patterns

### Builder Pattern

All components use the builder pattern for flexible, type-safe construction.

```rust
let reader = CsvItemReaderBuilder::<Product>::new()
    .has_headers(true)
    .delimiter(b',')
    .from_path("products.csv")?;

let step = StepBuilder::new("process-products")
    .chunk(100)
    .reader(&reader)
    .processor(&processor)
    .writer(&writer)
    .skip_limit(10)
    .build();
```

**Benefits:**
- Clear, readable API
- Compile-time validation
- Sensible defaults
- Flexible configuration

### Strategy Pattern

Readers, processors, and writers are interchangeable strategies.

```mermaid
graph LR
    Step[Step] --> IReader[ItemReader Trait]
    IReader --> CSV[CsvReader]
    IReader --> JSON[JsonReader]
    IReader --> DB[DatabaseReader]

    style IReader fill:#3b82f6,color:#fff
    style CSV fill:#10b981,color:#fff
    style JSON fill:#10b981,color:#fff
    style DB fill:#10b981,color:#fff
```

### Template Method Pattern

Job and Step execution follows a template with customizable steps.

```rust
// Framework provides the template
pub fn run(&self) -> Result<JobExecution, BatchError> {
    self.before_job()?;        // Hook
    let result = self.execute_steps()?;
    self.after_job()?;         // Hook
    Ok(result)
}
```

## Memory Model

### Chunk Processing Memory Usage

```mermaid
graph TB
    subgraph "Memory Usage Per Chunk"
        Input[Input Buffer<br/>~chunk_size items]
        Processing[Processing Buffer<br/>~chunk_size items]
        Output[Output Buffer<br/>~chunk_size items]
    end

    subgraph "Total Memory"
        Total[~3 × chunk_size × item_size]
    end

    Input --> Processing
    Processing --> Output
    Output --> Total

    style Total fill:#f59e0b,color:#fff
```

**Memory Optimization:**
- Adjust `chunk_size` based on available memory
- Use streaming for large items
- Paginate database queries
- Clear buffers after each chunk

### Resource Management

```rust
// Resources are automatically cleaned up
{
    let reader = CsvItemReaderBuilder::new()
        .from_path("large_file.csv")?;

    // File handle opened
    let step = StepBuilder::new("process")
        .chunk(1000)  // Only 1000 items in memory
        .reader(&reader)
        .build();

    job.run()?;
    // File handle automatically closed
}
```

## Concurrency Model

Spring Batch RS is designed for single-threaded execution by default, but supports parallelization strategies.

### Current Model: Sequential

```mermaid
sequenceDiagram
    participant J as Job
    participant S1 as Step 1
    participant S2 as Step 2
    participant S3 as Step 3

    J->>S1: Execute
    S1-->>J: Complete
    J->>S2: Execute
    S2-->>J: Complete
    J->>S3: Execute
    S3-->>J: Complete
```

### Future: Parallel Steps

```mermaid
graph TB
    Job[Job] --> Split{Split}
    Split --> Step1[Step 1]
    Split --> Step2[Step 2]
    Split --> Step3[Step 3]
    Step1 --> Join{Join}
    Step2 --> Join
    Step3 --> Join
    Join --> Next[Next Step]

    style Split fill:#3b82f6,color:#fff
    style Join fill:#10b981,color:#fff
```

## Transaction Management

### Database Transactions

```mermaid
sequenceDiagram
    participant Step
    participant Reader
    participant Writer
    participant DB

    Step->>DB: BEGIN TRANSACTION

    loop Each chunk
        Step->>Reader: read(chunk_size)
        Reader->>DB: SELECT...
        DB-->>Reader: rows
        Reader-->>Step: items

        Step->>Writer: write(items)
        Writer->>DB: INSERT/UPDATE...
    end

    Step->>DB: COMMIT

    Note over Step,DB: If error: ROLLBACK
```

### File Operations

File operations are not transactional by default. Use staging directories:

```rust
// Write to temporary location
let temp_writer = JsonItemWriterBuilder::<TempData>::new()
    .from_path("/tmp/output.json")?;

// On success, move to final location
std::fs::rename("/tmp/output.json", "/final/output.json")?;
```

## Error Handling Architecture

<Tabs>
  <TabItem label="Skip Strategy">
    ```rust
    let step = StepBuilder::new("fault-tolerant")
        .chunk(100)
        .reader(&reader)
        .processor(&processor)
        .writer(&writer)
        .skip_limit(10)  // Skip up to 10 errors
        .build();
    ```

    **Use when**: Individual item failures shouldn't stop the job
  </TabItem>

  <TabItem label="Fail-Fast Strategy">
    ```rust
    let step = StepBuilder::new("critical-process")
        .chunk(100)
        .reader(&reader)
        .processor(&processor)
        .writer(&writer)
        // No skip_limit - fail on first error
        .build();
    ```

    **Use when**: Data integrity is critical
  </TabItem>

  <TabItem label="Retry Strategy">
    ```rust
    struct RetryProcessor<P> {
        inner: P,
        max_retries: u32,
    }

    impl<I, O, P> ItemProcessor<I, O> for RetryProcessor<P>
    where
        P: ItemProcessor<I, O>,
    {
        fn process(&self, item: I) -> ItemProcessorResult<O> {
            let mut attempts = 0;
            loop {
                match self.inner.process(item.clone()) {
                    Ok(result) => return Ok(result),
                    Err(e) if attempts < self.max_retries => {
                        attempts += 1;
                        std::thread::sleep(Duration::from_millis(100 * attempts));
                    }
                    Err(e) => return Err(e),
                }
            }
        }
    }
    ```

    **Use when**: Transient failures are expected (network, locks)
  </TabItem>
</Tabs>

## Performance Characteristics

### Throughput vs Memory Trade-offs

```mermaid
graph LR
    Small[Small Chunks<br/>10-50 items] -->|Lower Memory<br/>Lower Throughput| Result1[Safe for<br/>Large Items]
    Medium[Medium Chunks<br/>100-500 items] -->|Balanced| Result2[Recommended<br/>Default]
    Large[Large Chunks<br/>1000+ items] -->|Higher Memory<br/>Higher Throughput| Result3[High-Performance<br/>Small Items]

    style Medium fill:#10b981,color:#fff
```

### Benchmarks (Typical)

| Operation | Small Chunks (10) | Medium Chunks (100) | Large Chunks (1000) |
|-----------|-------------------|---------------------|---------------------|
| CSV Read | 5,000/sec | 45,000/sec | 180,000/sec |
| JSON Write | 3,000/sec | 28,000/sec | 95,000/sec |
| DB Insert | 500/sec | 4,000/sec | 12,000/sec |

<Aside type="tip">
  **Optimization Tip**: Start with chunk size of 100, then increase based on memory availability and item size.
</Aside>

## Extension Points

### Custom ItemReader

```rust
use spring_batch_rs::core::item::ItemReader;
use spring_batch_rs::BatchError;

struct ApiItemReader {
    url: String,
    page: usize,
    buffer: Vec<Item>,
}

impl ItemReader<Item> for ApiItemReader {
    fn read(&mut self) -> ItemReaderResult<Item> {
        if self.buffer.is_empty() {
            // Fetch next page
            self.fetch_page()?;
        }

        Ok(self.buffer.pop())
    }
}
```

### Custom Tasklet

```rust
use spring_batch_rs::core::step::{Tasklet, StepExecution, RepeatStatus};

struct CleanupTasklet {
    directory: PathBuf,
    days_old: u32,
}

impl Tasklet for CleanupTasklet {
    fn execute(&self, execution: &StepExecution)
        -> Result<RepeatStatus, BatchError>
    {
        // Custom cleanup logic
        self.delete_old_files()?;
        Ok(RepeatStatus::Finished)
    }
}
```

## Best Practices

<CardGrid>
  <Card title="1. Size Your Chunks Wisely" icon="rocket">
    - Start with 100 items
    - Monitor memory usage
    - Adjust based on item size
    - Consider database batch limits
  </Card>

  <Card title="2. Handle Errors Gracefully" icon="warning">
    - Set appropriate skip limits
    - Log skipped items
    - Implement retry logic for transient errors
    - Use validation early
  </Card>

  <Card title="3. Optimize I/O" icon="setting">
    - Use buffered readers/writers
    - Batch database operations
    - Compress network transfers
    - Cache reference data
  </Card>

  <Card title="4. Monitor & Measure" icon="information">
    - Track execution times
    - Monitor memory usage
    - Log progress regularly
    - Profile critical paths
  </Card>
</CardGrid>

## Summary

Spring Batch RS architecture provides:

✅ **Separation of Concerns**: Clear separation between reading, processing, and writing
✅ **Flexibility**: Multiple processing models (chunk vs tasklet)
✅ **Extensibility**: Easy to add custom components
✅ **Reliability**: Built-in error handling and recovery
✅ **Performance**: Optimized for throughput and memory efficiency
✅ **Type Safety**: Rust's strong type system prevents common errors

## Next Steps

- [Processing Models](/processing-models/) - Deep dive into chunk vs tasklet
- [Item Readers & Writers](/item-readers-writers/overview/) - Explore all I/O options
- [Examples](/examples/) - See architecture in action
