# arrdb-core

Foundational type system for arrDb — the multi-dimensional dataset engine.

This crate defines the core data model that every other crate in the workspace depends on. It has **zero internal dependencies** and only one external dependency (`thiserror`).

## Module Structure

```
src/
├── lib.rs          Re-exports all public types
├── dtype.rs        Cell data types, scalar values
├── dimension.rs    Array axes (named, typed, chunked)
├── schema.rs       Array schema (dimensions + attributes)
├── dataset.rs      Dataset schema, coordinate variables, data variables
├── coord_index.rs  Coordinate index (label-to-index lookup)
├── chunk_coord.rs  Chunk grid addressing
└── error.rs        Error types
```

## Core Types

### `DatasetSchema` — Dataset Definition (Primary)

The primary user-facing data model. A Dataset is a collection of data variables sharing named dimensions and coordinate variables — the xarray.Dataset equivalent.

```rust
struct DatasetSchema {
    name: String,
    dimensions: Vec<Dimension>,      // shared named axes
    data_vars: Vec<DataVar>,         // the arrays with actual data
    coords: Vec<CoordinateVar>,      // label arrays for dimensions
}

struct DataVar {
    name: String,           // e.g., "temperature"
    dtype: DType,           // cell data type
    dim_indices: Vec<usize>, // which dimensions this variable spans
}

struct CoordinateVar {
    dim_name: String,          // must match a dimension name
    dtype: DType,              // Float64 for lat/lon, Timestamp for time
    monotonicity: Monotonicity, // enables binary search for .sel()-style lookup
    len: u64,                  // number of coordinate values
}
```

Coordinate variables provide real-valued labels (latitude in degrees, time as timestamps) for a dimension's integer indices. They enable label-based selection (`ds.sel(lat=45.0)` → `SELECT temp[lat(45.0)] FROM ds`) and nearest-neighbor lookup.

### `DType` — Cell Data Types

```rust
enum DType {
    Bool,
    Int8, Int16, Int32, Int64,
    UInt8, UInt16, UInt32, UInt64,
    Float32, Float64,
    Timestamp(TimeUnit),
    Struct(Vec<Field>),
}
```

Deliberately restricted to types that map to fixed-size SIMD-friendly representations. No variable-length strings, no complex numbers, no Python objects.

### `Dimension` — Array Axes

```rust
struct Dimension {
    name: String,       // "x", "y", "time"
    dtype: DType,       // coordinate type
    lo: i64, hi: i64,   // extent (inclusive)
    chunk_size: u64,    // cells per chunk on this axis
}
```

Dimensions are **integer-indexed** with uniform chunking. The `lo:hi:chunk_size` triple defines a regular, contiguous index space.

Note: this crate imposes **no dimension limit**. All types are `Vec`-based and work with any dimension count. The fixed 8-dimension limit is enforced only by `arrdb-buffer`'s chunk header format — see [Dimension Limits](#dimension-limits) below.

### `ArraySchema` — Array Definition (Internal)

Used internally by the execution engine. Each data variable in a `DatasetSchema` maps to an `ArraySchema` for query execution.

```rust
struct ArraySchema {
    name: String,
    dimensions: Vec<Dimension>,   // axes
    attributes: Vec<Field>,       // cell value fields (data variables)
}
```

An array's shape is defined by its dimensions. Each cell holds one value per attribute. Attributes are stored column-oriented within chunks (one buffer per attribute).

### `ChunkCoord` — Chunk Addressing

```rust
struct ChunkCoord(Vec<i64>);  // position in chunk grid
```

Addresses a chunk's location in the grid. For a 2-D array with 256x256 chunks, cell (500, 700) maps to chunk (1, 2).

### `ScalarValue` — Typed Constants

```rust
enum ScalarValue {
    Bool(bool), Int8(i8), ..., Float64(f64),
    Timestamp(TimeUnit, i64),
    Null(DType),
}
```

Used for literals, fill values, and expression constants throughout the system.

### `ArrDbError` — Error Types

Covers type errors, dimension errors, schema validation, storage I/O, query parsing, and more. All via `thiserror` derive.

## Relationship to xarray

arrDb's data model is designed to mirror xarray's Dataset/DataArray/coordinate concepts while executing via SIMD-accelerated SQL instead of Python. The `DatasetSchema` type is the direct equivalent of `xarray.Dataset`.

### Where They Align

| Concept | xarray | arrdb-core |
|---|---|---|
| Named dimensions | `dim="time"` | `Dimension { name: "time" }` |
| Dataset (collection of variables) | `Dataset` | `DatasetSchema` |
| Data variables | `DataArray` | `DataVar` + `ArraySchema` |
| Coordinate variables | `ds.coords['lat']` | `CoordinateVar` |
| Label-based selection | `ds.sel(lat=45.0)` | Coordinate resolution via `CoordinateIndex` |
| Nearest-neighbor lookup | `ds.sel(lat=45.3, method='nearest')` | `NEAREST(45.3)` in SQL subscripts |
| Typed cell values | NumPy dtypes | `DType` enum |
| Chunked execution | Dask chunks | `chunk_size` in `Dimension` |

### Where They Diverge

#### Index Model

**xarray** supports fully irregular coordinate arrays — a "time" dimension can carry labels like `[2020-01-01, 2020-01-15, 2020-03-07]` with arbitrary spacing. Coordinates are first-class data arrays.

**arrdb-core** uses integer-indexed dimensions with a regular extent (`lo:hi:chunk_size`). Coordinate variables provide real-valued labels (via `CoordinateVar` + `CoordinateIndex`), but the underlying index space is always a regular grid. This enables SIMD-friendly chunk addressing.

#### Alignment

**xarray** automatically aligns arrays by coordinate labels. **arrdb-core** matches by integer index position; alignment is explicit via SQL `JOIN` syntax.

#### Chunking

**xarray/Dask** treats chunking as optional; chunks can be irregular sizes. **arrdb-core** makes chunking mandatory and uniform — `chunk_size` is in the dimension definition. A chunk is the fundamental unit of storage, compression, I/O, and SIMD execution.

#### Metadata

**xarray** attaches rich `.attrs` dicts (CF-conventions). **arrdb-core** carries no general metadata — metadata lives in the chunk format layer (FlatBuffers in `arrdb-buffer`).

#### Data Types

**xarray/NumPy** supports complex numbers, variable-length strings, Python objects. **arrdb-core** restricts to fixed-size SIMD-friendly types (numeric + bool + timestamp + struct).

#### Dimension Limits

**xarray** has no dimension limit. **arrdb-core** also has no hard limit — all types are `Vec`-based. The 8-dimension cap exists only in `arrdb-buffer`'s chunk header (fixed `[u64; 8]` for zero-parse reads), covering ~99.9% of scientific workloads.

### Positioning

| | xarray | Rasdaman / TileDB / SciDB | arrdb-core |
|---|---|---|---|
| Primary concept | Dataset | Array | Dataset |
| Coordinate model | Label arrays | Integer grid | Integer grid + coordinate variables |
| Alignment | Automatic by label | Explicit | Explicit |
| Chunking | Optional overlay | Core primitive | Core primitive |
| Query interface | Python API | SQL / rasql | SQL with array extensions |
| Regularity | Irregular OK | Regular grid | Regular grid |

arrDb bridges the gap: it uses xarray's Dataset-first data model (named variables, coordinate variables, label-based selection) but executes via a database engine with SIMD kernels and SQL syntax.
