# Narsil

Narsil is a Rust inference serving engine. The engine is designed to be backend
agnostic: network I/O and request schema live in Narsil, while model runtimes
plug in behind a small backend trait.

The Rust package name is `narsil-engine`; downstream Rust code imports the
library as `narsil_engine`.

The supported model import targets are deliberately narrow:

- PyTorch artifacts
- ONNX artifacts

Users are expected to bring their own serving protobufs and Rust
preprocess/postprocess code when the public tensor schema is not enough for
their product API. Narsil provides the serving structure, scheduling, runtime
integration points, and validation boundaries.

The current implementation has moved the active serving path to:

- Tokio runtime
- tonic gRPC over HTTP/2
- optional HTTP/JSON gateway over HTTP/1.1
- feature-gated HTTP/3 over QUIC
- protobuf tensor schema generated with `tonic-prost-build`
- Burn/Flex CPU backends, and feature-gated Burn/CUDA backends (MLP and DLRM)
- a native-torch DLRM backend ("route A") that serves TorchRec TorchScript
  artifacts through libtorch + FBGEMM via `tch-rs` (`--features torch`)
- three execution modes: `inline`, an async worker pool, and a continuous-batching
  collector that coalesces concurrent requests into one fused forward

The Knafeh and PyO3-era Python scaffolding have been removed. Python clients
should use the official `grpcio` library against the public protobuf schema.

Full documentation — architecture, decision log, benchmarks, and the roadmap to
v1.0.0 — is published at **https://narsil.pages.dev** and authored under `docs/`
(VitePress).

## Repository Layout

```text
Narsil/
+-- Cargo.toml
+-- build.rs
+-- proto/narsil/inference/v1/inference.proto
+-- src/
|   +-- backend/      # InferenceBackend trait + impls
|   |   +-- burn.rs       # affine smoke backend
|   |   +-- mlp.rs        # Burn MLP backend
|   |   +-- dlrm.rs       # Burn DLRM backend (+ opt-in cuTile interaction)
|   |   `-- torch_dlrm.rs # native torch DLRM via libtorch/FBGEMM (feature `torch`)
|   +-- error.rs
|   +-- model.rs      # PyTorch/ONNX artifact and user adapter traits
|   +-- schema.rs     # Protobuf/native schema conversion and validation
|   +-- server.rs     # tonic service + inline / worker-pool / batch executors
|   `-- main.rs       # gRPC server entry point
+-- docs/             # VitePress documentation site (deployed to narsil.pages.dev)
+-- benchmarks/       # grpcio_vs_fastapi.py, torchrec_dlrm_compare.py
+-- examples/         # burn_cuda_mlp.rs, torch_dlrm_spike.rs, torch_dlrm_backend_smoke.rs
+-- third_party/torchrec  # submodule, build-time only (artifact generation)
+-- .github/workflows # rust-ci, publish-crate, deploy-docs
+-- tests/grpc_smoke.rs
`-- REPORTS.md        # Benchmark results (TorchRec vs Narsil; route A)
```

## Serving Architecture

```text
gRPC clients / HTTP JSON clients / HTTP/3 QUIC clients
    |
    |  protobuf / tonic, JSON / axum, or JSON / h3
    |  - PredictRequest
    |  - PredictResponse
    |  - typed Tensor messages
    v
Tokio transport layer
    |
    |  shared inference service
    |  - request validation
    |  - schema conversion
    |  - async worker-pool dispatch
    v
InferenceBackend trait
    |
    |  synchronous backend implementation
    |  executed on bounded inference workers
    v
Backend implementation
    |
    |  Burn affine / MLP / DLRM backends (Flex CPU; CUDA with --features cuda)
    |  Native torch DLRM via libtorch + FBGEMM (tch-rs) with --features torch
    |  ONNX Runtime, TensorRT, candle, or custom runtimes later
    v
PredictResponse tensors
```

## Model Import And User Serving Logic

Narsil's built-in import surface should stay focused on PyTorch and ONNX. We
should not grow first-party import paths for arbitrary framework formats unless
they are translated through one of those two forms.

The extension point for product-specific serving logic is
`UserServingAdapter<WireRequest, WireResponse>`:

```rust
pub trait UserServingAdapter<WireRequest, WireResponse>: Send + Sync + 'static {
    fn model(&self) -> &ModelArtifact;
    fn preprocess(&self, request: WireRequest) -> Result<InferenceRequest>;
    fn postprocess(&self, response: InferenceResponse) -> Result<WireResponse>;
}
```

This lets a downstream service define its own generated tonic service from its
own `.proto` files, parse product-specific request fields in Rust, build a
Narsil `InferenceRequest`, and convert the `InferenceResponse` back into its
own wire response. Narsil only needs to guarantee the general serving structure:
backend loading, request validation, worker scheduling, and error boundaries.

Model artifacts are represented with:

```rust
ModelArtifact::pytorch("ranker", "models/ranker.pt");
ModelArtifact::onnx("detector", "models/detector.onnx");
```

## Wire Schema

The default wire protocol is standard gRPC with protobuf messages:

- `Inference/Predict`
- `PredictRequest`
- `PredictResponse`
- `Tensor`
- `DataType`

The protobuf definition lives at:

```text
proto/narsil/inference/v1/inference.proto
```

Tensor payloads are compact binary byte buffers with explicit dtype and shape:

```protobuf
message Tensor {
  string name = 1;
  DataType dtype = 2;
  repeated uint64 shape = 3;
  bytes data = 4;
}
```

The Rust schema layer validates dtype, shape, and byte size before dispatching
to a backend. Services with their own protos can use this schema internally
after their own Rust preprocess step.

## HTTP/JSON Gateway

gRPC remains the default transport. Set `NARSIL_HTTP_ADDR` to expose an HTTP/JSON
gateway alongside the gRPC listener without changing the backend or executor:

```bash
NARSIL_HTTP_ADDR=127.0.0.1:8080 cargo run -- 127.0.0.1:50051
```

Endpoints:

- `POST /v1/infer` accepts one request and returns one response.
- `POST /v1/infer_batch` accepts `{ "requests": [...] }` and returns per-item
  `{ "status": "ok", "response": ... }` or `{ "status": "error", "error": ... }`.

The JSON tensor shape mirrors the protobuf tensor schema. `data` is base64-encoded
little-endian tensor bytes, and `dtype` is one of `F32`, `F64`, `I64`, `I32`, `U8`,
or `Bool`:

```json
{
  "model_name": "burn-affine",
  "inputs": [
    {
      "name": "input",
      "dtype": "F32",
      "shape": [2],
      "data": "AACAPwAAAEA="
    }
  ],
  "parameters": {}
}
```

The gateway also exposes the health routes described below.

## QUIC / HTTP/3

Build with `--features quic` and set `NARSIL_QUIC_ADDR` to expose the same JSON
routes over HTTP/3 on QUIC:

```bash
NARSIL_QUIC_ADDR=127.0.0.1:50052 cargo run --features quic -- 127.0.0.1:50051
```

The QUIC transport serves the same paths as the HTTP/JSON gateway:

- `POST /v1/infer`
- `POST /v1/infer_batch`
- `GET /metrics`
- `GET /health/live`
- `GET /health/ready`

QUIC is opt-in at both compile time and runtime. If `NARSIL_QUIC_ADDR` is set
without the `quic` feature, startup fails fast with a configuration error before
any listener binds.

### TLS

The QUIC transport always uses TLS 1.3 and advertises ALPN `h3`. Supply a real,
PEM-encoded certificate chain and private key via environment variables for any
non-local deployment:

```bash
NARSIL_QUIC_TLS_CERT=/etc/narsil/tls/cert.pem \
NARSIL_QUIC_TLS_KEY=/etc/narsil/tls/key.pem \
NARSIL_QUIC_ADDR=0.0.0.0:443 cargo run --features quic -- 0.0.0.0:50051
```

Both variables must be set together; setting only one is a configuration error.
If **neither** is set, the server generates an **ephemeral self-signed
certificate for local development only** and logs a warning. Such a certificate
cannot be verified by clients and **must not be used in production** — always
provide `NARSIL_QUIC_TLS_CERT`/`NARSIL_QUIC_TLS_KEY` there.

## Observability

Set `NARSIL_OBSERVABILITY_ADDR` to expose `/metrics` and the health probes on a
**dedicated listener** that you should **bind privately** (e.g. `127.0.0.1` or a
sidecar-only interface) — `/metrics` can leak request volume/timing, so it is kept off
the public inference port by design:

```bash
NARSIL_OBSERVABILITY_ADDR=127.0.0.1:9090 cargo run -- 127.0.0.1:50051
```

Endpoint placement:

- **`NARSIL_OBSERVABILITY_ADDR` (recommended):** serves `/metrics`, `/health/live`,
  `/health/ready`. Bind it privately.
- **`NARSIL_HTTP_ADDR` (JSON gateway):** serves the inference routes plus
  `/health/live` and `/health/ready`. `/metrics` is **not** mounted here when an
  observability listener is configured. If `NARSIL_HTTP_ADDR` is set *without*
  `NARSIL_OBSERVABILITY_ADDR`, `/metrics` is mounted on the gateway as a single-port
  convenience (a startup warning is logged) — bind the gateway privately in that case.
- **gRPC-only (neither set):** metrics are still collected but there is no HTTP endpoint
  to scrape them; a startup warning is logged. Set `NARSIL_OBSERVABILITY_ADDR` to expose
  them.

Endpoints:

- `GET /metrics` renders Prometheus text exposition.
- `GET /health/live` returns `204` while the process and its execution machinery are
  alive (e.g. the continuous-batching collector loop has not died), `503` otherwise.
- `GET /health/ready` returns `204` once the backend is loaded (valid `metadata()`) and
  the executor is live, and `503` during shutdown/drain or before the backend is ready.

Metric names:

- `narsil_requests_total{transport,endpoint,status}` — request counter.
- `narsil_request_latency_seconds{transport,endpoint,status}` — latency histogram.
- `narsil_in_flight_requests{transport,endpoint}` — in-flight gauge.
- `narsil_batch_size{transport,endpoint}` — **client-supplied** batch size, i.e. the
  number of requests in a single transport call (1 for unary `infer`/`predict`).
- `narsil_fused_batch_size` — the **effective fused batch** the continuous-batching
  collector forms before calling `infer_batch` on the backend (the batch the GPU
  actually sees). Only recorded in `batch` execution mode.

Current transport label values are `grpc`, `http`, and `quic`; endpoint values include
`predict`, `infer`, and `infer_batch`.

## Backend API

Backends implement `InferenceBackend`:

```rust
pub trait InferenceBackend: Send + Sync + 'static {
    fn metadata(&self) -> BackendMetadata;
    fn infer(&self, request: InferenceRequest) -> Result<InferenceResponse>;
}
```

Backend implementations are deliberately synchronous. The gRPC service can run
them inline for debugging or dispatch them to a bounded async worker pool so
Tokio network I/O tasks do not execute blocking inference directly.

Worker-pool execution is controlled with:

```bash
NARSIL_EXECUTION_MODE=worker
NARSIL_INFERENCE_WORKERS=1
NARSIL_INFERENCE_QUEUE_DEPTH=1024
```

`NARSIL_EXECUTION_MODE=inline` keeps the old direct execution path for
debugging. CUDA serving defaults to one worker; CPU serving defaults to up to
four workers unless overridden.

`NARSIL_EXECUTION_MODE=batch` enables the continuous-batching collector: it
accumulates concurrent requests up to `NARSIL_BATCH_SIZE` (or until
`NARSIL_BATCH_TIMEOUT_MS` elapses) and runs them through a backend's `infer_batch`
as one fused forward. For a single GPU the winning configuration is one execution
lane plus batch mode — one large fused forward beats many small contending ones.

## Burn Backends

`BurnBackend` is the small affine backend used for schema and transport smoke
tests. It accepts an F32 input tensor of rank 1 through 4 and returns an F32
output tensor after a simple affine Burn tensor operation.

The smoke test configures `BurnBackend::with_affine(2.0, 1.0)` and verifies:

```text
[1.0, 2.0, 3.0, 4.0] -> [3.0, 5.0, 7.0, 9.0]
```

`BurnMlpBackend` is the first real inference workload. It uses deterministic
weights and runs a two-layer MLP with ReLU:

```text
input_dim -> hidden_dim -> output_dim
```

The default benchmark shape is:

```text
1024 -> 2048 -> 1000
```

The CUDA serving path is feature-gated:

```bash
cargo run --release --features cuda -- 127.0.0.1:50051
```

with:

```bash
NARSIL_BACKEND=burn-cuda-mlp
NARSIL_MLP_INPUT_DIM=1024
NARSIL_MLP_HIDDEN_DIM=2048
NARSIL_MLP_OUTPUT_DIM=1000
NARSIL_EXECUTION_MODE=worker
NARSIL_INFERENCE_WORKERS=1
```

The standalone CUDA demo is:

```bash
cargo run --release --features cuda --example burn_cuda_mlp -- \
  --batch 1 --input 1024 --hidden 2048 --output 1000 --warmup 5 --iters 20
```

If CUDA fails with `CUDA_ERROR_STUB_LIBRARY`, make sure the NVIDIA driver
library path appears before CUDA toolkit stub paths:

```bash
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/local/cuda-13.0/lib64 \
  target/release/examples/burn_cuda_mlp \
  --batch 1 --input 1024 --hidden 2048 --output 1000
```

## Native Torch DLRM (Route A)

The `torch-dlrm` backend (feature `torch`) serves a TorchRec DLRM TorchScript
artifact through libtorch via `tch-rs`, running the real FBGEMM table-batched
embedding and interaction kernels. It implements `infer_batch`, so `batch` mode
coalesces requests into one fused forward. On an L4, serving the same INT8 artifact
it matches the TorchRec C++ baseline's single-request latency and exceeds its
throughput via continuous batching (see the benchmarks page).

Build against an installed libtorch 2.11 (`tch` 0.24), e.g. a `torch==2.11` env:

```bash
LIBTORCH_USE_PYTORCH=1 cargo build --release --features torch
```

Generate the current-stack TorchRec INT8 DLRM artifact from the parked submodule:

```bash
micromamba run -n narsil python -m pip install click iopath pyre-extensions tensordict
micromamba run -n narsil python scripts/create_torchrec_dlrm_artifact.py
```

These are pure-Python helpers that the parked `third_party/torchrec` imports at
build time only (they are not engine runtime deps). They are intentionally left
unpinned so pip can resolve versions compatible with the `narsil` env's torch
2.11, but keep them on whatever versions stay compatible with torch 2.11 — do not
let them pull in a build that conflicts with the strict torch↔fbgemm ABI pairing
(torch 2.11 + fbgemm-gpu 1.6.0). If a fresh install ever drags in an incompatible
`tensordict`/`pyre-extensions`, pin it explicitly to the last torch-2.11-compatible
release.

Run it:

```bash
NARSIL_BACKEND=torch-dlrm \
NARSIL_TORCH_MODEL_PATH=target/torchrec/dlrm_int8_seed0_torch211_fbgemm16.pt \
NARSIL_TORCH_PRELOAD_LIBS="<fbgemm .so set>" \
NARSIL_CUDA_DEVICE=0 NARSIL_EXECUTION_MODE=batch NARSIL_BATCH_SIZE=16 \
  target/release/narsil-engine 127.0.0.1:50051
```

`NARSIL_TORCH_PRELOAD_LIBS` is an ordered, comma-separated list of `.so` files
dlopened (RTLD_GLOBAL) at startup for FBGEMM plugin registration, so the artifact's
`fbgemm::*` ops resolve. CUDA libtorch builds retain `libtorch_cuda.so` at link time.
The architecture and decision-log pages at https://narsil.pages.dev cover the full
rationale.

## Development

Build and test the root Narsil engine:

```bash
cargo check
cargo test
```

Run the gRPC server:

```bash
cargo run -- 127.0.0.1:50051
```

The server defaults to `127.0.0.1:50051` when no address is provided.

The Burn affine workload used by the benchmark can be configured with:

```bash
NARSIL_BURN_SCALE=2.0 NARSIL_BURN_BIAS=1.0 cargo run --release -- 127.0.0.1:50051
```

Run the MLP workload on Burn/Flex CPU:

```bash
NARSIL_BACKEND=burn-mlp cargo run --release -- 127.0.0.1:50051
```

## Python grpcio Compatibility

Narsil's gRPC API is standard tonic/prost protobuf over HTTP/2, so Python
clients should use the official `grpcio` package. The compatibility and
throughput benchmark generates Python stubs from `proto/`, starts the Rust
server, starts a FastAPI REST baseline, and drives concurrent AsyncIO clients:

```bash
uv run benchmarks/grpcio_vs_fastapi.py --requests 500 --warmup 50 --concurrency 32 --elements 1024
```

Run the real MLP workload benchmark against the CPU backend:

```bash
uv run benchmarks/grpcio_vs_fastapi.py \
  --workload mlp \
  --requests 200 \
  --warmup 20 \
  --concurrency 16 \
  --input-dim 1024 \
  --hidden-dim 2048 \
  --output-dim 1000 \
  --execution-mode worker \
  --inference-workers 4
```

Run it against the CUDA backend:

```bash
uv run benchmarks/grpcio_vs_fastapi.py \
  --workload mlp \
  --requests 500 \
  --warmup 100 \
  --concurrency 16 \
  --input-dim 1024 \
  --hidden-dim 2048 \
  --output-dim 1000 \
  --rust-backend burn-cuda-mlp \
  --execution-mode worker \
  --inference-workers 1
```

The benchmark writes `REPORTS.md` by default.

## Release

Rust CI runs on pull requests and pushes to `main`. It checks formatting,
clippy, and tests with the locked dependency graph.

Releasing is tag/Release-driven and does two things: publishes `narsil-engine`
to crates.io and deploys the docs to Cloudflare Pages. Make sure `Cargo.toml` has
the matching `version`, then either push the version tag:

```bash
git tag v0.4.0
git push origin v0.4.0
```

or publish a GitHub Release for it (`gh release create v0.4.0 --generate-notes`).
Both workflows trigger on a `v*.*.*` **tag push** *and* on a **published GitHub
Release** — a Release that creates its tag does not emit a tag-push event, so both
triggers are wired:

- `publish-crate.yml` verifies the tag matches `Cargo.toml`, re-runs the Rust
  checks, does `cargo publish --dry-run --locked`, then publishes with
  `CARGO_REGISTRY_TOKEN` (skipping if the version is already on crates.io).
- `deploy-docs.yml` POSTs the Cloudflare Pages deploy hook
  (`CLOUDFLARE_DEPLOY_HOOK_URL`), which rebuilds and deploys the docs site.

## Migration Notes

- The active communication layer is tonic + Tokio.
- The active schema codec is protobuf generated by tonic/prost.
- Backend dispatch is backend agnostic through `InferenceBackend`.
- Burn/Flex and feature-gated Burn/CUDA MLP workloads are wired.
- ONNX Runtime can be reintroduced as another `InferenceBackend`.
- Dedicated async inference workers are wired. CPU-core affinity and device
  buffer reuse are still future tuning work.
