# SER-001: Block Serialization with Dictionary Compression

## Summary

Full L2 blocks are serialized through a two-stage pipeline: bincode serialization followed by zstd dictionary compression. The write path transforms `L2Block` -> `bincode::serialize()` -> `zstd::compress_with_dictionary(dict, level=3)` -> `CF_BLOCKS`. The read path reverses this exactly: `CF_BLOCKS` -> `zstd::decompress_with_dictionary(dict)` -> `bincode::deserialize()` -> `L2Block`. This achieves a typical 3-5x compression ratio, significantly reducing disk I/O and storage requirements.

## Specification

```rust
impl BlockStore {
    /// Serialize and compress a full block for storage in CF_BLOCKS.
    /// Uses bincode for structural serialization, then zstd with a trained
    /// dictionary for compression (level 3).
    pub fn serialize_block(&self, block: &L2Block) -> Result<Vec<u8>> {
        let raw = bincode::serialize(block)
            .map_err(|e| BlockStoreError::Serialization(e.to_string()))?;

        match &self.zstd_dictionary {
            Some(dict) => {
                let mut compressor = zstd::bulk::Compressor::with_dictionary(
                    self.compression_level,
                    dict,
                )?;
                Ok(compressor.compress(&raw)?)
            }
            None => {
                // Fallback: plain zstd when no dictionary is loaded
                Ok(zstd::encode_all(&raw[..], self.compression_level)?)
            }
        }
    }

    /// Decompress and deserialize a full block from CF_BLOCKS bytes.
    /// Attempts dictionary decompression first; falls back to plain zstd
    /// for blocks stored before dictionary training.
    pub fn deserialize_block(&self, compressed: &[u8]) -> Result<L2Block> {
        let raw = match &self.zstd_decompressor {
            Some(decompressor) => {
                match decompressor.decompress(compressed, self.max_block_size) {
                    Ok(bytes) => bytes,
                    Err(_) => {
                        // Fallback: plain zstd for pre-dictionary blocks
                        zstd::decode_all(compressed)?
                    }
                }
            }
            None => zstd::decode_all(compressed)?,
        };

        bincode::deserialize(&raw)
            .map_err(|e| BlockStoreError::Serialization(e.to_string()))
    }
}
```

## Acceptance Criteria

1. `serialize_block` produces bytes that are valid zstd-compressed data.
2. `deserialize_block(serialize_block(block))` returns an `L2Block` equal to the original.
3. When a dictionary is loaded, compression uses the dictionary (verifiable via zstd frame header).
4. When no dictionary is loaded, plain zstd compression is used as fallback.
5. Dictionary decompression failure triggers fallback to plain zstd decompression.
6. Compression level defaults to 3.
7. Compressed output is typically 3-5x smaller than the bincode-serialized input for representative blocks.

## Implementation Notes

- The compression level (default 3) balances speed and ratio; higher levels yield marginally better ratios but significantly slower compression.
- `max_block_size` caps decompressed output to prevent memory exhaustion from malformed data.
- The dictionary is trained separately (see SER-005) and shared across all serialization calls.
- Pre-dictionary blocks (stored before training) use plain zstd and must remain readable after dictionary is loaded, hence the fallback path.

## Test Plan

- **Unit: round-trip** -- Serialize a block with `serialize_block`, deserialize with `deserialize_block`, verify equality.
- **Unit: compression ratio** -- Serialize representative blocks, measure ratio, assert within 2-6x range.
- **Unit: dictionary compression** -- With dictionary loaded, verify compressed output contains dictionary ID in zstd frame header.
- **Unit: plain zstd fallback** -- With no dictionary loaded, verify serialization/deserialization still works.
- **Unit: dictionary decompression fallback** -- Compress a block with plain zstd, then deserialize with dictionary loaded; verify fallback succeeds.
- **Unit: corrupted input** -- Pass invalid bytes to `deserialize_block`, verify `BlockStoreError::Serialization` returned.

## Expected Test Files

- `tests/ser_001_tests.rs` (flat `[[test]]` crate per repo convention; houses all SER-001 scenarios: round-trip, ratio, dictionary/plain/fallback)
