---
id: PRD-0014
title: "M13 — server observability: spans, OTel export, structured stdout"
status: done
owner: "Aaron Roney"
created: 2026-07-03
updated: 2026-07-03

principles:
- "stdout stays the primary surface: human-readable, structured, greppable — OTLP export is additive, never a replacement."
- "Instrument the request paths, not everything: auth, join, fan-out, admin ops, store round-trips."
- "Zero-cost when disabled: no OTLP endpoint configured → no exporter task, no per-message overhead beyond the existing tracing events."

references:
- name: "tracing-opentelemetry"
  url: https://docs.rs/tracing-opentelemetry
- name: "OpenTelemetry Rust SDK (OTLP exporter)"
  url: https://docs.rs/opentelemetry-otlp

acceptance_tests:
- id: uat-001
  name: "Server request paths emit spans with session-path / channel fields (asserted via a capturing subscriber)"
  command: cargo make test
  uat_status: verified
- id: uat-002
  name: "With no OTLP endpoint configured, serve runs exactly as today (stdout fmt only)"
  command: cargo make test
  uat_status: verified

tasks:
- id: T-001
  title: "Span coverage on the server: handshake/auth, session frame dispatch, fan-out, admin ops, store calls"
  priority: 1
  status: done
  notes: "Today there are ~6 bare events and zero #[instrument] spans in the server path. Add #[tracing::instrument] with useful fields (session path, channel, frame kind) and consistent levels."
- id: T-002
  title: "OTLP export behind env config (CONCLAVE_OTLP_ENDPOINT), layered with the stdout fmt subscriber"
  priority: 2
  status: done
  notes: "tracing-opentelemetry + opentelemetry-otlp, tokio batch exporter; deps land with this milestone per the constitution. Resource attrs: service.name=conclave, service.version."
- id: T-003
  title: "Structured stdout option (CONCLAVE_LOG_FORMAT=json|pretty) for platform log pipelines"
  priority: 2
  status: done
  notes: "fmt layer already exists; add the json formatter switch (Fly/Loki-friendly), keep pretty as the default. RUST_LOG / -v semantics unchanged."
- id: T-004
  title: "Docs: observability section in DESIGN + deploy notes (fly.toml env, sample RUST_LOG)"
  priority: 3
  status: done
  notes: "Document the env surface and what each span/level means for an operator."
---

# Summary

Bring the server's telemetry up to the standard of Aaron's other services: real span coverage on
the request paths, an optional OTLP exporter configured purely by env, and a JSON stdout format
switch for platform log pipelines — with the current human-readable stdout as the unchanged
default.

# Problem

The server today has a `tracing-subscriber` fmt layer (`RUST_LOG` / `-v`) but only ~6 bare events
across the whole serve path and zero spans — a production incident (like the 2026-07-03 supersede
storm) is diagnosed from client-side symptoms because the server emits almost nothing about auth
outcomes, session lifecycle, or fan-out. There is also no OTel story: nothing to point a collector
at, and stdout is not machine-parseable.

# Goals

1. An operator can answer "who connected, what failed, and why" from server telemetry alone.
2. `CONCLAVE_OTLP_ENDPOINT=…` lights up traces in any OTLP collector with zero code changes.
3. Log pipelines get JSON when asked; humans keep the pretty format.

# Technical Approach

Layered `tracing-subscriber` registry: the existing fmt layer (pretty or JSON by
`CONCLAVE_LOG_FORMAT`), plus a `tracing-opentelemetry` layer added only when
`CONCLAVE_OTLP_ENDPOINT` is set (batch exporter over tonic/http). `#[tracing::instrument]` on the
session driver, hub verbs, and store round-trips with skip-by-default args and explicit fields
(`session_path`, `channel`, `frame`). Levels: info for lifecycle (established/superseded/reaped),
warn for refusals, debug for per-frame dispatch. The subscriber init stays in `bin.rs` (the bridge
deliberately keeps stdout clean — MCP owns it; its telemetry goes to stderr as today).

# Assumptions

- OTLP deps (opentelemetry, opentelemetry-otlp, tracing-opentelemetry) land with this milestone
  (constitution: dependencies arrive with the milestone that first needs them).
- Fly.io deploy consumes stdout logs as-is; a collector endpoint is optional and external.

# Constraints

- The bridge's stdout is the MCP transport — the bridge is out of scope beyond keeping its stderr
  behavior; this PRD is the *server*.
- No per-message allocation on the fan-out hot path when no subscriber is interested (use
  `tracing`'s disabled-fast-path idioms; fields, not format-eager strings).

# References to Code

- `src/bin.rs` (subscriber init, `log_directive`), `src/server/session.rs` / `src/server/hub.rs`
  (the paths to instrument), `src/server/wss.rs` (accept/upgrade lifecycle), `fly.toml` (env).

# Non-Goals (MVP)

- Metrics (counters/histograms) and a /metrics endpoint — traces + logs first; metrics can ride
  the same OTel pipeline later.
- Client/bridge-side tracing export.
- Log persistence or shipping infrastructure (that's the platform's job).

# History

- 2026-07-03: Drafted from live-testing feedback ("are we emitting good otel on the server?" —
  today: fmt subscriber only, ~6 events, no spans, no exporter).

- 2026-07-03: All four tasks landed: instrumented request paths (kind-only fields, no bodies\/tokens in telemetry), env-gated OTLP\/HTTP export verified against a live fake collector, CONCLAVE_LOG_FORMAT=json, and docs\/fly.toml notes. Gate green (201 tests).
