# Embeddenator-VSA Development Context

**Repo**: embeddenator-vsa  
**Version**: 0.20.0-alpha.1  
**Focus**: Sparse ternary VSA primitives, SIMD optimizations, balanced ternary encoding  
**Last Updated**: 2026-01-16

---

## Active Technologies

- **Rust** 1.84+ (stable with nightly for SIMD)
- **rand** 0.9.2: Random vector generation
- **serde/serde_json**: Serialization
- **criterion** 0.8.1: Benchmarking
- **libc** 0.2.180: System calls

### SIMD Support
- **AVX2** (x86_64): Feature-gated behind `simd` flag
- **NEON** (aarch64): Feature-gated behind `simd` flag
- **Scalar fallback**: Always available

---

## Project Structure

```
embeddenator-vsa/
├── specs/                      # Feature specifications (SDD)
├── src/
│   ├── lib.rs                 # Public API exports
│   ├── vsa.rs                 # SparseVec + VSA operations (bundle, bind, cosine)
│   ├── simd_cosine.rs         # SIMD-optimized cosine similarity
│   ├── codebook.rs            # Reversible encoding/decoding
│   ├── ternary.rs             # Ternary logic (trits, bytes)
│   ├── ternary_vec.rs         # PackedTritVec (bit-packed representation)
│   └── dimensional.rs         # Dimensional ternary operations
├── benches/
│   ├── vsa_ops.rs             # Bundle, bind, thin benchmarks
│   └── simd_cosine.rs         # SIMD vs scalar comparisons
├── tests/
│   └── codebook_roundtrip_tests.rs  # Reconstruction validation
├── REFACTORING_ANALYSIS.md    # P1 refactor plan (bundle_hybrid, SIMD, errors)
├── GAP_ANALYSIS.md            # Implementation gaps
├── IMPLEMENTATION_PLAN.md     # Detailed roadmap
└── docs/
    ├── ARCHITECTURE.md         # Design decisions
    ├── SIMD_DESIGN.md          # SIMD strategy
    ├── BITSLICED_TERNARY_DESIGN.md
    └── TERNARY_REPRESENTATION_GUIDE.md
```

---

## Commands

### Build
```bash
# Standard
cargo build --release

# With SIMD (requires AVX2 or NEON)
cargo build --release --features simd

# With balanced ternary Phase 2
cargo build --release --features bt-phase-2
```

### Test
```bash
# All tests (30 unit + 11 integration)
cargo test --all

# With SIMD
cargo test --features simd

# Single test
cargo test test_bundle_associative
```

### Benchmark
```bash
# VSA operations
cargo bench --bench vsa_ops

# SIMD comparisons
cargo bench --bench simd_cosine
```

---

## Core API

### SparseVec
```rust
use embeddenator_vsa::{SparseVec, ReversibleVSAConfig};

// Create sparse ternary vector
let config = ReversibleVSAConfig::new(10000, 100);
let vec = SparseVec::random(&config);

// VSA operations
let bundled = vec1.bundle(&vec2);           // Superposition (⊕)
let bound = vec1.bind(&vec2);               // Composition (⊙)
let sim = vec1.cosine_similarity(&vec2);    // Similarity measure
let thinned = vec.thin(50);                 // Reduce non-zero count
```

### Codebook (Reversible Encoding)
```rust
use embeddenator_vsa::Codebook;

let codebook = Codebook::new(10000, 10);
let encoded = codebook.encode_data(b"hello world");
let decoded = codebook.decode(encoded)?;
```

---

## Critical Refactoring Needed (REFACTORING_ANALYSIS.md)

### P1: Complete bundle_hybrid_many()
**Current**: Incomplete with TODO comments; immediately falls back to scalar
**Fix**: Implement proper heuristic selector for bundling strategies

### P1: Real AVX2/NEON Implementation
**Current**: `simd_cosine.rs` immediately breaks to scalar fallback
**Issue**: "For now, fall back to scalar for the vectorized portion due to complexity"
**Fix**: Implement proper AVX2 loop for sorted merge intersection

### P1: Unified Error Handling
**Current**: Mix of Option, panic!, and silent failures
**Fix**: Create `VsaError` enum; replace panic! with Result types

### P2: PackedTritVec Integration
**Current**: Declared but not integrated; only used with `bt-phase-2` feature
**Fix**: Benchmark and document performance characteristics; consider making default for dense vectors

### P2: Codebook Projection Heuristics
**Current**: Magic numbers (`0.3` threshold, `take(4)`)
**Fix**: Move to config struct; document rationale; add adaptive thresholding

---

## Code Conventions

### Style
- Follow Rust API Guidelines
- Zero clippy warnings
- Run `cargo fmt` before commit

### Error Handling (Planned)
```rust
pub enum VsaError {
    InvalidDimension { expected: usize, got: usize },
    InsufficientSparsity { min: f64, actual: f64 },
    EncodingFailure(String),
    DecodingFailure(String),
}
```

### SIMD
- Always provide scalar fallback
- Feature-gate SIMD behind `simd` flag
- Document platform requirements (AVX2 = x86_64 with CPU support, NEON = aarch64)

---

## Current Focus (2026-01-16)

### Recent Updates (main/dev branches)
- 26 commits pulled from remote
- GAP_ANALYSIS.md, IMPLEMENTATION_PLAN.md added
- Comprehensive docs: ARCHITECTURE.md, SIMD_DESIGN.md, USAGE_GUIDE.md
- Examples: basic_operations.rs, encoding_decoding.rs, similarity_search.rs
- Tests: codebook_roundtrip_tests.rs

### Next Tasks
1. Complete `bundle_hybrid_many()` implementation
2. Implement real AVX2/NEON loops (no scalar fallback)
3. Create `VsaError` and refactor error handling
4. Benchmark `PackedTritVec` vs `SparseVec`
5. Externalize codebook projection constants

---

## Recent Changes

### 2026-01-16: Spec-Kit Integration
- Added `.copilot` with repo context
- Created `specs/` directory structure
- Synced to main/dev branches (26 commits)

### 2026-01-15: Documentation & Examples
- Created comprehensive architecture docs
- Added 3 working examples
- Established codebook roundtrip tests

### 2026-01-10: Dependabot Updates
- Upgraded rand 0.8.5 → 0.9.2
- Upgraded criterion 0.5.1 → 0.8.1
- Removed broken bincode 3.0 dependency

---

## Agents

Specialized agents in `.github/agents/`:
- `vsa-mathematician.agent.md`: VSA theory and operations
- `simd-optimizer.agent.md`: SIMD implementations and benchmarking
- `rust-implementer.agent.md`: Idiomatic Rust patterns

---

<!-- MANUAL ADDITIONS START -->
<!-- MANUAL ADDITIONS END -->
