# Knafeh E2E Testing SOP

End-to-end testing procedure for the Knafeh QUIC/HTTP3 RPC framework. Tests mirror production traffic patterns and generate simulation workloads that exercise both transport backends.

## Running Tests

```bash
cargo test --no-default-features                           # All Rust tests
cargo test --no-default-features --test e2e_test           # Core + streaming
cargo test --no-default-features --test e2e_resilience_test # Retries + errors
cargo test --no-default-features --test e2e_load_test      # Load simulation
cargo test --no-default-features --test e2e_bench_test     # Codec + server comparison
cargo test --no-default-features --release --test e2e_perf_test -- --ignored --nocapture  # High-volume perf

PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 maturin develop && pytest tests/test_e2e_python.py -v  # Python E2E
```

Each test spins up an isolated server on an ephemeral port (`127.0.0.1:0`) with runtime-generated TLS certs (`rcgen`). Shared test infrastructure lives in `tests/common/mod.rs` (EchoService, TestCerts, helpers).

---

## Test Scenarios

### A: Unary RPC — Core Path

| # | Scenario | Production analog | Workload | Key assertions |
|---|----------|-------------------|----------|----------------|
| A1 | Basic echo | Single RPC happy path | 1 small JSON request | Status Ok, body echoed |
| A2 | JSON roundtrip | Codec fidelity | 1 structured JSON request | Parsed response matches expected shape |
| A3 | Empty body | Health check / ping | 1 `{}` request | Status Ok, body preserved |
| A4 | Large payload | Bulk data transfer | 1 request, ~1MB | Body length preserved, no truncation |
| A5 | Multiple services | Microservice routing | 2 requests to different services | Each routed correctly |
| A6 | Method not found | Stale client calling removed endpoint | 1 request to unknown path | Status NotFound |

### B: Metadata & Middleware

| # | Scenario | Production analog | Workload | Key assertions |
|---|----------|-------------------|----------|----------------|
| B1 | Metadata roundtrip | Trace ID propagation | 1 request with custom metadata | Key/value preserved in response |
| B2 | Client interceptor | Auth token injection | 1 request through interceptor | Injected metadata visible on server |
| B3 | Interceptor chain | Logging + auth + tracing stacked | 1 request through 3 interceptors | All metadata present, order correct |
| B4 | Interceptor rejection | Auth rejects unauthenticated request | 1 request, interceptor returns Err | Error returned, service never called |
| B5 | Server-side interceptor | Server logging / rate limiting | 1 request, server interceptor modifies response | Server-injected metadata in response |

### C: Connection Pooling & Reuse

| # | Scenario | Production analog | Workload | Key assertions |
|---|----------|-------------------|----------|----------------|
| C1 | Sequential reuse | Steady-state traffic | 5 sequential requests | Pool size stays at 1 |
| C2 | Concurrent burst | Traffic spike | 10 concurrent requests | All succeed, pool size <= 4 |

### D: Server-Streaming RPC

| # | Scenario | Production analog | Workload | Key assertions |
|---|----------|-------------------|----------|----------------|
| D1 | Basic stream | Event feed delivering N items | 5 JSON chunks | All received in order |
| D2 | Empty stream | Query returning no results | 0 chunks | Stream ends immediately |
| D3 | Early cancellation | Client navigates away mid-stream | Request 10000 chunks, read 3, drop | No panic, no leak |
| D4 | Mixed unary + streaming | Dashboard: query + live subscription | 1 unary + 1 stream, same connection | No interference |
| D5 | Concurrent streams | Multiple subscriptions | 5 parallel streams, 10 chunks each | All 50 chunks correct |
| D6 | Large stream chunks | Streaming large records | 5 chunks of 100KB+ each | Data intact, no framing errors |

### E: Retry & Resilience

| # | Scenario | Production analog | Workload | Key assertions |
|---|----------|-------------------|----------|----------------|
| E1 | Retry on Unavailable | Transient overload | Unavailable for first 2 calls, then Ok | Client retries and succeeds |
| E2 | Non-retryable error | Bad request | InvalidArgument returned | No retry, immediate error |
| E3 | Retry exhaustion | Persistent outage | Always Unavailable, max_retries=2 | Error after 3 attempts |
| E4 | Backoff timing | Exponential backoff verification | Timed retry sequence | Each wait longer than previous |

### F: Error Handling & Edge Cases

| # | Scenario | Production analog | Workload | Key assertions |
|---|----------|-------------------|----------|----------------|
| F1 | Codec error | Malformed request from buggy client | Non-JSON bytes | InvalidArgument status |
| F3 | Service error propagation | Business logic error | Service returns PermissionDenied | Correct status + message |

### G: Load Simulation

| # | Scenario | Production analog | Workload | Key assertions |
|---|----------|-------------------|----------|----------------|
| G1 | Sustained throughput | Steady traffic | 100 sequential requests | All succeed |
| G2 | Burst concurrency | Traffic spike | 50 concurrent requests | All succeed |
| G3 | Mixed workload | Real traffic (70% unary, 20% stream, 10% error) | 100 concurrent mixed | All complete, no deadlock |
| G4 | Slow service under load | Degraded backend | 20 concurrent to 100ms-latency service | All complete, pool not exhausted |

### H: Transport Comparison (perf benchmark, `#[ignore]`)

| # | Scenario | Workload | Key metrics |
|---|----------|----------|-------------|
| H1 | Raw QUIC sequential | 10K calls | rpc/s, p50/p90/p99 |
| H2 | Raw QUIC parallel x32 | 5K calls | rpc/s |
| H3 | HTTP/3 + Protobuf sequential | 10K calls | rpc/s, p50/p90/p99 |
| H4 | HTTP/3 + JSON sequential | 10K calls | rpc/s, p50/p90/p99 |

---

## Test Service Catalog

| Service | Method | Kind | Behavior | Used by |
|---------|--------|------|----------|---------|
| **EchoService** | `echo/echo` | Unary | Returns body unchanged | A1-A6, C1-C2, G1-G2, H1-H4 |
| **GreeterService** | `greeter/greet` | Unary | `{"name":"X"}` -> `{"message":"Hello, X!"}` | A2, A5 |
| **MetadataEchoService** | `metadata_echo/echo` | Unary | Copies request metadata to response | B1-B5 |
| **CountService** | `count/count` | ServerStreaming | Streams N chunks `{"i":0}..{"i":N-1}` | D1-D6, G3 |
| **SlowService** | `slow/call` | Unary | Sleeps configurable duration, then echoes | G4 |
| **FailService** | `fail/call` | Unary | Returns error for first N calls, then succeeds | E1-E4 |
| **ErrorService** | `error/*` | Unary | Returns the named RPC status code | F1, F3 |

Shared infrastructure (EchoService, TestCerts, helpers) lives in `tests/common/mod.rs`.

---

## Verification Requirements

Every E2E test must assert:

1. **Status code** — Expected RPC status (Ok, NotFound, etc.)
2. **Body integrity** — Response body matches expected content
3. **No panics** — Test exits cleanly, no `unwrap` on unexpected `Err`
4. **Cleanup** — Streams dropped, connections released

Global invariants:

- **Connection reuse** — Pool size does not grow unboundedly
- **No hanging** — All tests complete within 30s
- **No leaked tasks** — Background tasks terminate when streams/connections close
- **Isolated** — Each test uses its own server on a unique port
- **Idempotent** — Tests run in any order, any number of times

---

## Test File Organization

```
tests/
├── common/mod.rs            # Shared: EchoService, TestCerts, tls_server(), start_test_server()
├── e2e_test.rs              # Categories A-D (core + streaming)
├── e2e_resilience_test.rs   # Categories E-F (retries, errors)
├── e2e_load_test.rs         # Category G (concurrency, load)
├── e2e_bench_test.rs        # Codec + server comparison benchmark
├── e2e_perf_test.rs         # Category H (high-volume perf, #[ignore])
├── integration_test.rs      # Unit-level component tests
├── test_knafeh.py           # Python binding smoke tests
└── test_e2e_python.py       # Python E2E (async + sync, protobuf codec)
```

---

## Adding New Tests

1. Pick the category (A-H) that fits
2. Reuse `EchoService`, `TestCerts`, `tls_server()`, `start_test_server()` from `common/mod.rs`
3. Follow the pattern: `start_test_server()` -> `test_client()` -> exercise -> assert
4. Assert specifically (status, body, metadata, pool state) not just "no error"
5. Document the production scenario in a comment:

```rust
/// Simulates: dashboard making both a query (unary) and subscribing
/// to live updates (streaming) on the same QUIC connection.
#[tokio::test]
async fn test_e2e_mixed_unary_and_streaming() { ... }
```
