# Compatibility Testing Framework

This directory contains the **dual-server compatibility testing** infrastructure
for validating athena-auth against the canonical better-auth (TypeScript)
implementation.

## Architecture

```mermaid
graph TD
    CompatFramework --> SpecAndContractSuites
    CompatFramework --> DualServerTests
    CompatFramework --> SnapshotAndProperty
    SpecAndContractSuites --> OpenAPISchema
    DualServerTests --> RustServer
    DualServerTests --> NodejsRefServer
```

## Components

### 1. Endpoint Spec Validation (`tests/compat_endpoint_tests.rs`)

Automated tests that parse `better-auth.yaml` and validate Rust responses
against the OpenAPI spec.

```bash
# Run all endpoint spec validations
cargo test --test compat_endpoint_tests -- --nocapture

# Run only the main endpoint sweep
cargo test --test compat_endpoint_tests test_spec_driven_endpoint_validation -- --nocapture
```

Features:

- **Schema auto-validation**: Parses OpenAPI spec schemas and validates response fields
- **`$ref` resolution**: Follows `$ref` references to component schemas
- **camelCase enforcement**: Ensures all field names use camelCase (not snake_case)
- **Route coverage analysis**: Reports what % of spec endpoints are implemented
- **Response type signatures**: Generates human-readable type docs for all endpoints
- **Cross-endpoint consistency**: Validates user/session objects are consistent across responses
- **Shape comparison engine**: Compares two JSON structures ignoring dynamic values

### 2. Coverage and Contract Validation (`tests/compat_coverage_tests.rs`, `tests/compatibility_tests.rs`)

Contract tests and route coverage reports against the reference spec.

### 3. Field/Shape/Consistency Validation (`tests/compat_field_tests.rs`, `tests/compat_consistency_tests.rs`, `tests/compat_unit_tests.rs`)

CamelCase field checks, shared object consistency checks, and shape/unit validation helpers.

### 4. Plugin-Specific Coverage (`tests/compat_admin_tests.rs`, `tests/compat_organization_tests.rs`, `tests/compat_passkey_tests.rs`)

Focused compatibility coverage for admin, organization, and passkey endpoints.

### 5. Reference Server (`compat-tests/reference-server/`)

A Node.js server running the canonical athena-auth (TypeScript) implementation
for dual-server comparison testing.

#### Setup

```bash
cd compat-tests/reference-server
npm install
npm start  # starts on port 3100
```

#### Usage

The reference server is used by the dual-server test runner to:

1. Send identical requests to both implementations
2. Compare response shapes (not exact values)
3. Report any structural differences

## Test Categories

| Category         | Command                                                                                   | Description                                    |
| ---------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------- |
| Endpoint spec sweep | `cargo test --test compat_endpoint_tests -- --nocapture` | OpenAPI-driven endpoint shape validation |
| Endpoint coverage report | `cargo test --test compat_coverage_tests test_route_coverage_analysis -- --nocapture` | Reports which spec endpoints are implemented |
| camelCase check | `cargo test --test compat_field_tests test_all_responses_use_camel_case -- --nocapture` | Ensures frontend-compatible field names |
| Error shapes | `cargo test --test compat_endpoint_tests test_error_response_shapes_match_spec -- --nocapture` | Validates error response format |
| Flow consistency | `cargo test --test compat_consistency_tests test_auth_flow_user_object_consistency -- --nocapture` | Validates user object consistency across flows |
| Type signatures | `cargo test --test compat_field_tests test_response_type_signatures -- --nocapture` | Generates response type documentation |
| Contract suite | `cargo test --test compatibility_tests -- --nocapture` | Route coverage + contract tests |
| Admin suite | `cargo test --test compat_admin_tests -- --nocapture` | Admin endpoint compatibility |
| Organization suite | `cargo test --test compat_organization_tests -- --nocapture` | Organization endpoint compatibility |
| Passkey suite | `cargo test --test compat_passkey_tests -- --nocapture` | Passkey endpoint compatibility |
| Snapshot suite | `cargo test --test snapshot_tests -- --nocapture` | Response snapshot regression |
| Property suite | `cargo test --test property_tests -- --nocapture` | Property-based invariants |
| Dual-server suite | `cargo test --test dual_server_tests -- --nocapture` | Rust vs reference-server structural parity |
| HTTP dual-server suite | `cargo test --test server_http_dual_tests -- --nocapture` | End-to-end HTTP parity |
| Dual-server latency benchmark | `cargo test --test dual_server_benchmark_tests -- --nocapture` | Per-endpoint Rust vs reference response-time report with repeated-run aggregation |

## Quick Start Commands

Run from repo root (`athena-auth/`).

```bash
# Full compatibility suite (same groups as CI compat + P2 tests)
cargo test --test compat_endpoint_tests -- --nocapture
cargo test --test compat_field_tests -- --nocapture
cargo test --test compat_coverage_tests -- --nocapture
cargo test --test compat_consistency_tests -- --nocapture
cargo test --test compat_unit_tests -- --nocapture
cargo test --test compat_admin_tests -- --nocapture
cargo test --test compat_organization_tests -- --nocapture
cargo test --test compat_passkey_tests -- --nocapture
cargo test --test snapshot_tests -- --nocapture
cargo test --test property_tests -- --nocapture
cargo test --test dual_server_tests -- --nocapture
cargo test --test server_http_dual_tests -- --nocapture
# Benchmark latency and generate report
cargo test --test dual_server_benchmark_tests -- --nocapture
```

WSL note:

- On Windows + WSL, prefer `bash scripts/run_dual_server_benchmark_wsl.sh`.
- That script bootstraps a Linux Node runtime in WSL, rebuilds `better-sqlite3` for that runtime, runs the benchmark, and then renders the HTML dashboard.
- This avoids the Windows `node.exe` networking boundary where the spawned reference server can appear healthy but still be unreachable from the WSL Rust test process.
- The WSL runner defaults to a more extensive profile than the bare Rust test:
  - `DUAL_SERVER_BENCH_ITERATIONS=20`
  - `DUAL_SERVER_BENCH_WARMUP=3`
  - `DUAL_SERVER_BENCH_REPEATS=3`

Benchmark report output:

- `compat-tests/reports/dual-server-latency-report.md`
- `compat-tests/reports/dual-server-latency-report.json`
- `compat-tests/reports/dual-server-latency-report.html`
- Optional env overrides:
  - `DUAL_SERVER_BENCH_ITERATIONS` (default `12`)
  - `DUAL_SERVER_BENCH_WARMUP` (default `2`)
  - `DUAL_SERVER_BENCH_REPEATS` (default `1`)
  - `ATHENA_BENCH_ARGON2_MEMORY_COST`
  - `ATHENA_BENCH_ARGON2_TIME_COST`
  - `ATHENA_BENCH_ARGON2_PARALLELISM`

Rendered report notes:

- Athena Auth is rendered in indigo (`#3440D5`).
- Better Auth is rendered in gray (`#6B7280`).
- The HTML dashboard includes grouped latency charts, delta charts, a per-run breakdown grid, and a benchmark-flow diagram.

## Command Migration (Old -> Current)

The old target `spec_driven_compat_tests` was split into focused suites:

- `spec_driven_compat_tests` -> `compat_endpoint_tests`
- `spec_driven_compat_tests test_route_coverage_analysis` -> `compat_coverage_tests test_route_coverage_analysis`
- `spec_driven_compat_tests test_all_responses_use_camel_case` -> `compat_field_tests test_all_responses_use_camel_case`
- `spec_driven_compat_tests test_error_response_shapes_match_spec` -> `compat_endpoint_tests test_error_response_shapes_match_spec`
- `spec_driven_compat_tests test_auth_flow_user_object_consistency` -> `compat_consistency_tests test_auth_flow_user_object_consistency`
- `spec_driven_compat_tests test_response_type_signatures` -> `compat_field_tests test_response_type_signatures`

## Adding New Tests

### Adding a new spec-driven endpoint test

Add a new section to the `test_spec_driven_endpoint_validation` test:

```rust
// --- POST /your-endpoint ---
let (status, body) = send_request(
    &auth,
    post_json_with_auth("/your-endpoint", serde_json::json!({...}), &token),
).await;
validator.validate_endpoint("/your-endpoint", "post", status, &body);
```

The validator will automatically:

1. Look up the endpoint in `better-auth.yaml`
2. Extract the expected response schema
3. Validate all required fields are present
4. Check field types match
5. Verify camelCase naming

### Adding a new shape comparison test

```rust
let diffs = compare_shapes(&reference_json, &target_json, "", false);
assert!(diffs.is_empty(), "Shape mismatch: {:?}", diffs);
```
