# scirs2-neural TODO

## Status: v0.4.3 Released (May 3, 2026)

## v0.3.3 Completed

### Attention Mechanisms
- Rotary Position Embeddings (RoPE)
- Grouped Query Attention (GQA)
- Linear attention
- Efficient attention
- Sparse attention
- Multi-head latent attention

### Mixture of Experts
- Top-k routing with load balancing
- Expert capacity and auxiliary loss
- MoE transformer block integration

### Capsule Networks
- Dynamic routing between capsules
- Squash activation
- EM routing variant

### Spiking Neural Networks (SNN)
- Leaky Integrate-and-Fire (LIF) neurons
- Spike-Timing Dependent Plasticity (STDP)
- Rate coding and temporal coding

### Graph Neural Networks (GNN)
- Graph Convolutional Networks (GCN)
- Graph Attention Networks (GAT)
- GraphSAGE
- Graph Isomorphism Network (GIN)
- Message Passing Neural Networks
- DiffPool and SAGPool graph pooling
- Global add/mean/max pooling

### Vision Architectures
- SWIN Transformer (shifted window self-attention)
- Vision Transformer (ViT) with patch embeddings
- UNet encoder-decoder
- CLIP dual-encoder (vision + text)
- ConvNeXt (Tiny, Small, Base, Large, XLarge)
- PatchEmbedding module

### NLP / Sequence Architectures
- GPT-2 causal language model
- T5 encoder-decoder
- Full transformer (encoder + decoder)
- Positional encodings: sinusoidal, learned, RoPE, relative

### Generative Models
- Generative Adversarial Networks (GAN)
- Variational Autoencoders (VAE)
- Diffusion models (DDPM)
- Normalizing flow models
- Energy-based models

### Training Infrastructure
- Knowledge distillation (response-based and feature-based)
- Continual learning (EWC)
- Meta-learning (MAML-style)
- Contrastive learning (SimCLR, MoCo)
- Multitask learning
- Self-supervised pretraining
- Magnitude-based and structured pruning
- Post-training quantization and QAT
- DPO (Direct Preference Optimization)
- PPO for RLHF
- Reward modeling and preference data
- Gradient checkpointing
- Half-precision (FP16) training utilities

### Serialization
- Model graph serialization format
- Portable weight format (versioned)

### Compression
- Model compression utilities
- On-device optimization

## v0.4.2 / v0.4.3 — Completed

### State Space Models — Implemented in v0.4.2
- [x] Mamba/SSM architecture (selective state space model) in src/models/architectures/mamba.rs
  - `MambaConfig` with builder methods (d_model, d_state, d_conv, expand, n_layers, vocab_size, num_classes)
  - `SelectiveSSM` (S6 selective scan with ZOH discretization)
  - `MambaBlock` (Conv1D causal convolution, SiLU gating, residual connection)
  - `Mamba` full model (Layer trait impl, optional classifier head, final LayerNorm)
  - `S4Layer` (non-selective SSM with HiPPO initialization)
  - 10 tests passing (config, creation, forward, classifier, numerical stability, conv1d, SSM, S4, block)

### Wave 44 Additions (v0.4.2)
- [x] NAS module repaired and re-enabled in lib.rs (74 tests)
- [x] CMA-ES optimizer (10 tests)
- [x] Enhanced BPE tokenizer + chat templates (14 tests)
- [x] Pipeline parallelism + tensor parallelism wired into training infrastructure
- [x] Numerical validation tests (40)
- [x] Cross-crate consistency tests (16)

## v0.4.3 Status (2026-05-03)

- Workspace version bumped to 0.4.3; all crate dependency strings updated.
- ~2,786 `#[test]` functions across `src/` and `tests/`.
- cargo check + clippy: clean (no warnings, no errors)
- No-unwrap policy: PASS (no production violations)

## v0.4.0 Roadmap

### Attention — Implemented in v0.4.0
- [x] Flash Attention v2 (tiled memory-efficient attention)
- [x] Multi-query attention (MQA)
- [x] Grouped Query Attention (GQA)

### Quantization — Implemented in v0.4.0
- [x] INT4 weight-only quantization (src/quantization/int4.rs — group-quantized, nibble-packed)
- [x] INT8 activation quantization (src/quantization/int8.rs)
- [x] GPTQ-style post-training quantization (src/quantization/gptq.rs)

### Export and Interop — Implemented in v0.4.0
- [x] ONNX-like model export (src/export/onnx.rs — pure-Rust, oxicode serialization)
- [x] Weight conversion utilities for interop with other frameworks (src/export/weights.rs)

### Efficient Fine-Tuning — Implemented in v0.4.0
- [x] LoRA (Low-Rank Adaptation) (src/lora/linear.rs)
- [x] Adapter layers (src/lora/adapter.rs — bottleneck adapters with optional residual)
- [x] Prefix tuning (src/layers/prefix_tuning.rs — reparameterized prefix tokens)

### Distributed Training — Implemented in v0.4.0
- [x] Gradient compression (TopK sparsification, PowerSGD) (src/training/gradient_compression.rs)
- [x] Pipeline parallelism (src/training/pipeline_parallel.rs — GPipe FThenB + 1F1B schedules)
- [x] Tensor parallelism primitives (src/training/tensor_parallel.rs — column/row parallel + parallel embedding)

### Architecture Search — Implemented in v0.4.0, repaired in v0.4.2
- [x] Neural Architecture Search (NAS) integration (src/nas/ — ENAS, multi-objective, hardware-aware)
- [x] Differentiable NAS (DARTS/GDAS/SNAS) (src/nas/gdas.rs, src/nas/snas.rs)
- [x] NAS module re-enabled in lib.rs (Wave 44, v0.4.2): truncated source files repaired, 74 tests passing

## Known Issues / Technical Debt

- Some doc tests are marked `#[ignore]` pending API stabilization
- WASM target requires additional feature gating for large model weights
- GPU acceleration stubs exist in `hardware/` but require `scirs2-core::gpu` completion
