# SER-005: Dictionary Training and Management

## Summary

The blockstore manages a zstd compression dictionary lifecycle: on first startup with no dictionary, it operates in plain-zstd mode. After accumulating 1000 stored blocks, it trains a dictionary from a random sample of block bodies using `zstd::dict::from_samples()`. The trained dictionary (~100KB) is persisted to `CF_METADATA` under the key `META_ZSTD_DICT`. On subsequent startups, the dictionary is loaded from metadata. If dictionary decompression fails for a given block, the store falls back to plain zstd to handle pre-dictionary blocks transparently.

## Specification

```rust
const DICT_TRAINING_THRESHOLD: u64 = 1000;
const DICT_TARGET_SIZE: usize = 100 * 1024; // ~100KB
const META_ZSTD_DICT: &str = "zstd_dict";

impl BlockStore {
    /// Load the zstd dictionary from CF_METADATA on startup.
    /// Returns None if no dictionary has been trained yet.
    fn load_dictionary(&self) -> Result<Option<Vec<u8>>> {
        self.db
            .get_cf(&self.cf_metadata, META_ZSTD_DICT.as_bytes())
            .map_err(Into::into)
    }

    /// Train a zstd dictionary from a random sample of stored block bodies.
    /// Called automatically when block_count crosses DICT_TRAINING_THRESHOLD.
    fn train_dictionary(&self) -> Result<Vec<u8>> {
        // Sample random blocks from CF_BLOCKS for training
        let samples = self.sample_block_bodies(DICT_TRAINING_THRESHOLD as usize)?;

        let dict = zstd::dict::from_samples(&samples, DICT_TARGET_SIZE)
            .map_err(|e| BlockStoreError::Serialization(
                format!("dictionary training failed: {e}")
            ))?;

        // Persist dictionary to CF_METADATA
        self.db.put_cf(
            &self.cf_metadata,
            META_ZSTD_DICT.as_bytes(),
            &dict,
        )?;

        Ok(dict)
    }

    /// Check if dictionary training should be triggered after a put.
    /// Called internally by the write path.
    fn maybe_train_dictionary(&self) -> Result<()> {
        if self.zstd_dictionary.is_some() {
            return Ok(()); // Already have a dictionary
        }

        let count = self.block_count()?;
        if count >= DICT_TRAINING_THRESHOLD {
            let dict = self.train_dictionary()?;
            self.set_dictionary(dict)?;
        }

        Ok(())
    }

    /// Initialize dictionary state on startup.
    pub fn init_dictionary(&mut self) -> Result<()> {
        if let Some(dict_bytes) = self.load_dictionary()? {
            self.set_dictionary(dict_bytes)?;
        }
        // If no dictionary, operate in plain-zstd mode
        Ok(())
    }
}
```

## Acceptance Criteria

1. On fresh startup with no dictionary in `CF_METADATA`, the store MUST operate in plain-zstd mode (no dictionary decompressor initialized).
2. After storing the 1000th block, the store MUST trigger dictionary training.
3. Dictionary training MUST call `zstd::dict::from_samples()` with a sample of block bodies.
4. The trained dictionary MUST be persisted to `CF_METADATA` under the key `META_ZSTD_DICT`.
5. On subsequent startup, `init_dictionary` MUST load the dictionary from `CF_METADATA` and initialize the compressor/decompressor.
6. Blocks stored before the dictionary was trained MUST remain readable (dictionary decompression fails, falls back to plain zstd).
7. The dictionary size MUST be approximately 100KB (target size parameter).

## Implementation Notes

- Dictionary training is a one-time operation per database; once trained, the dictionary is stable unless explicitly retrained.
- The sample for training should be randomly selected, not just the first 1000 blocks, to capture representative byte patterns.
- `zstd::dict::from_samples()` expects a flat `Vec<Vec<u8>>` of sample data; each sample should be the raw bincode-serialized block body (pre-compression).
- The fallback mechanism in `deserialize_block` (SER-001) handles the mixed-mode window where some blocks are plain-zstd and newer blocks are dictionary-compressed.
- Consider logging when dictionary training occurs and when fallback decompression is triggered, for operational visibility.

## Test Plan

- **Integration: fresh startup plain-zstd** -- Create a new BlockStore, verify no dictionary loaded, store and retrieve a block using plain zstd.
- **Integration: training trigger at 1000** -- Store 1000 blocks, verify dictionary training triggers, verify dictionary bytes present in `CF_METADATA`.
- **Integration: dictionary persistence** -- Train dictionary, close store, reopen, verify dictionary loaded from metadata.
- **Integration: pre-dictionary blocks readable** -- Store blocks in plain-zstd mode, train dictionary, verify pre-dictionary blocks still readable via fallback.
- **Unit: dictionary size** -- Train a dictionary, verify size is within expected range (50KB-150KB).
- **Unit: no double training** -- After dictionary is loaded, verify `maybe_train_dictionary` is a no-op.
- **Integration: mixed-mode reads** -- Store blocks before and after dictionary training, verify all blocks readable in a single session.

## Expected Test Files

- `tests/ser_005_tests.rs`
