# sphereQL

[![CI](https://github.com/bkahan/sphereQL/actions/workflows/ci.yml/badge.svg)](https://github.com/bkahan/sphereQL/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Crates.io](https://img.shields.io/crates/v/sphereql.svg)](https://crates.io/crates/sphereql)
[![PyPI](https://img.shields.io/pypi/v/sphereql.svg)](https://pypi.org/project/sphereql/)

**Project high-dimensional embeddings onto a 3D sphere for fast semantic
search, spatial queries, category-aware exploration, and interactive
visualization.**

sphereQL maps vectors from any embedding model (OpenAI, Cohere,
sentence-transformers, etc.) onto spherical coordinates via one of four
projection families — linear PCA, kernel PCA with a Gaussian (RBF)
kernel, Laplacian eigenmap over a k-NN similarity graph, or
UMAP-on-sphere (Adam-optimized in the tangent bundle of S², with an
optional supervised category term) — then indexes them with
shell/sector partitioning for fast nearest-neighbor lookups. A Category Enrichment Layer computes
inter-category relationships, classifies bridges (`Genuine` /
`OverlapArtifact` / `Weak`), and builds inner spheres for
high-resolution within-category search. sphereQL auto-tunes its
pipeline per corpus against a scalar `QualityMetric`; a meta-model
recalls winning configs from past tuner runs when a new corpus arrives.
Callable from Rust, Python, or the browser via WASM.

## How sphereQL compares to vector databases

sphereQL is **not** a drop-in replacement for FAISS, Qdrant, or
pgvector. Those systems return *the most similar vectors*. sphereQL
returns *the most similar vectors plus a 3D coordinate system you can
visualize, navigate, and reason about geometrically*.

| | FAISS | Qdrant / Milvus / Weaviate | pgvector | **sphereQL** |
|---|---|---|---|---|
| ANN nearest-neighbor | yes | yes | yes | yes (via 3D sphere index) |
| Hosted service | no | yes | yes (Postgres) | no — embed as library |
| Filtering on metadata | manual | yes | yes (SQL) | yes |
| 3D positions for visualization | no | no | no | **yes** |
| Spatial queries (cone, cap, shell, wedge) | no | no | no | **yes** |
| Category bridges + inner spheres | no | no | no | **yes** |
| Auto-tuned per corpus | no | no | no | **yes** |
| Returns full-d cosine accuracy | yes | yes | yes | only via hybrid re-rank |

**Use a vector DB when** you need scale-out k-NN over millions of
vectors, durable hosted storage, or multi-tenant access control.
**Use sphereQL when** you need 3D layouts for visualization, spatial
queries beyond k-NN, category-aware exploration, or an in-process
embedded library that runs in the browser via WASM.

The two compose: sphereQL ships [Pinecone and
Qdrant](docs/architecture.md) backends in `sphereql-vectordb` so you
can keep authoritative vectors in a hosted store and use sphereQL for
projection, layout, and category enrichment.

## Documentation

Full documentation lives under [`docs/`](docs/README.md).

- [Quickstart — Rust](docs/quickstart-rust.md) · [Python](docs/quickstart-python.md) · [WASM](docs/quickstart-wasm.md)
- [Architecture](docs/architecture.md) — workspace crates and feature flags
- [Projections](docs/projections.md) — how the four projection families work
- [Auto-tuning & meta-learning](docs/auto-tuning.md) — the metalearning framework
- [Lingua pipeline](sphereql-lingua/README.md) — text → `ConceptGraph` (six-stage Rust pipeline)
- [Empirical findings](docs/empirical-findings.md) — when does each projection win?
- [Examples catalog](docs/examples.md) · [Performance](docs/performance.md) · [Project status](docs/project-status.md)

## Install

```bash
# Rust
cargo add sphereql --features full
```

or pin it in `Cargo.toml`:

```toml
[dependencies]
sphereql = { version = "0.3.0", features = ["full"] }
```

```bash
# Python
pip install sphereql
```

See [architecture.md](docs/architecture.md) for feature-flag details.

## Rust — minimal example

```rust
use sphereql::embed::*;

// 1. Build a pipeline from categorized embeddings.
let input = PipelineInput {
    categories: vec![
        "science".into(), "science".into(),
        "cooking".into(), "cooking".into(),
    ],
    embeddings: vec![
        vec![0.1, 0.9, 0.3, 0.0],
        vec![0.2, 0.8, 0.4, 0.1],
        vec![0.9, 0.1, 0.0, 0.5],
        vec![0.8, 0.2, 0.1, 0.4],
    ],
};
let pipeline = SphereQLPipeline::new(input).unwrap();

// 2. Query nearest neighbors.
let query = PipelineQuery { embedding: vec![0.15, 0.85, 0.35, 0.05] };
let results = pipeline.query(SphereQLQuery::Nearest { k: 3 }, &query).unwrap();
```

See the [Rust quickstart](docs/quickstart-rust.md) for spatial indexing,
the layout engine, GraphQL, and the full embedding pipeline.
[`auto-tuning.md`](docs/auto-tuning.md) covers the `PipelineConfig` +
`auto_tune` + `MetaModel` workflow end-to-end.

## Python — minimal example

```python
import sphereql

categories = ["science", "science", "cooking", "cooking"]
embeddings = [
    [0.1, 0.9, 0.3, 0.0],
    [0.2, 0.8, 0.4, 0.1],
    [0.9, 0.1, 0.0, 0.5],
    [0.8, 0.2, 0.1, 0.4],
]

pipeline = sphereql.Pipeline(categories, embeddings)
results = pipeline.nearest([0.15, 0.85, 0.35, 0.05], k=3)

# Interactive 3D visualization in your browser
sphereql.visualize(categories, embeddings, title="My Embeddings")
```

The Python bindings cover the full Rust surface — PCA, Kernel PCA,
Laplacian eigenmap, UMAP-on-sphere (via `config={"projection_kind":
"UmapSphere"}` or `auto_tune`), `MetaModel`, `FeedbackAggregator`,
and the category enrichment layer. Type stubs (`.pyi`) are
auto-generated via `pyo3-stub-gen`. See the
[Python quickstart](docs/quickstart-python.md) for semantic search,
3D visualization, vector database bridges, and the core type
surface.

## WASM — minimal example

```bash
cd sphereql-wasm && wasm-pack build --target web
```

```javascript
import init, { Pipeline } from './pkg/sphereql_wasm.js';
await init();

const pipeline = new Pipeline(JSON.stringify({
  categories: ["science", "cooking"],
  embeddings: [[0.1, 0.9, 0.3], [0.9, 0.1, 0.0]],
}));
const results = pipeline.nearest(JSON.stringify([0.15, 0.85, 0.35]), 1);
```

Same bindings coverage as Python. Every pipeline / category /
metalearning method returns typed values via `tsify` — `wasm-pack build`
emits a `.d.ts` with named interfaces, no `JSON.parse` required. See
the [WASM quickstart](docs/quickstart-wasm.md) for category enrichment
in the browser.

## Workspace layout

| Crate | Role |
|---|---|
| `sphereql` | Umbrella crate with feature flags for selective imports. |
| `sphereql-core` | Spherical math — points, conversions, distance metrics, region types. |
| `sphereql-index` | Spatial indexing with shell + sector partitioning. |
| `sphereql-layout` | Layout engines (Fibonacci spiral, k-means, force-directed). |
| `sphereql-embed` | Projections, query pipeline, Category Enrichment Layer, metalearning framework. |
| `sphereql-graphql` | async-graphql schema: spatial queries (cone/shell/band/wedge), the full category enrichment surface, subscriptions, and a pluggable `TextEmbedder` trait for text query input. |
| `sphereql-vectordb` | Vector store bridge (InMemory, Qdrant, Pinecone) with hybrid search. |
| `sphereql-python` | Python bindings via PyO3/maturin. |
| `sphereql-wasm` | WASM bindings via wasm-bindgen. |
| `sphereql-corpus` | Shared example corpora: 775-concept built-in (31 academic domains) and 300-concept stress corpus, plus bulk-ingested parquet corpora from DBpedia (500K) and Wikidata (50K). |
| `sphereql-lingua` | Six-stage text → `ConceptGraph` pipeline that places every concept at a SphereQL `(r, θ, φ)` position. |
| `sphereql-examples` | Runnable examples across the whole workspace (not published). |
| `scripts/check-drift` | CI tool: fails when a new `sphereql-embed`/`sphereql-layout` public item isn't bound in Python/WASM (or allowlisted). |
| `scripts/check-versions` | CI tool: fails when any release-version string across manifests, READMEs, and docs disagrees with the canonical workspace/pyproject version. |
| `scripts/check-docs` | CI tool: fails when a workspace crate is missing from these tables, or a stated test-count floor no longer holds. |
| `scripts/check-doc-snippets` | CI tool: compile-checks the `rust` code blocks in the prose docs against `sphereql --features full`. |

Full dependency graph and crate-by-crate description in
[architecture.md](docs/architecture.md).

## Project status

sphereQL is at **v0.3.0**. The core API is functional and covered
by 850+ Rust tests plus 200+ Python binding tests, but may change
before 1.0. Known limitations and roadmap
are in [project-status.md](docs/project-status.md).

Binding parity is protected by a drift check (`scripts/check-drift`) —
new public items in `sphereql-embed` / `sphereql-layout` must either
have a Python/WASM binding or an allowlist entry with a reason in
`.bindings-ignore.toml`.

## Contributing

Fork, branch from `main`, and open a PR. Before pushing, run the core
verification trio:

```bash
cargo test --workspace --all-features --exclude sphereql-python
cargo clippy --workspace --all-features --all-targets -- -D warnings
cargo fmt --all -- --check
```

The codebase uses Rust 2024 edition; all CI checks must pass before
merge. See **[CONTRIBUTING.md](CONTRIBUTING.md)** for the full workflow,
the complete local command list, and the CI pipeline.

## License

[MIT](LICENSE)
