# Local observability stack

This folder gives `athena-auth` a predictable local place to inspect request traces, wide request events, and metrics without guessing which terminal or browser tab has the useful signal.

## What it gives you

- Loki on `http://localhost:3100` for structured request logs.
- Prometheus on `http://localhost:9090` for the server's `/metrics` endpoint.
- Grafana on `http://localhost:3002` with pre-provisioned Loki and Prometheus datasources.
- A pre-provisioned dashboard at `http://localhost:3002/d/athena-auth-local-observability/athena-auth-local-observability`.
- A local server config at [config.local-observability.yaml](config.local-observability.yaml) that turns on:
  - `observability.loki.enabled=true`
  - `observability.logging.format=json`
  - `observability.logging.wide_events=true`

## Start it

1. Start Docker Desktop or another local Docker engine.
2. Run [scripts/start-local-observability.ps1](../../../scripts/start-local-observability.ps1). If Docker Desktop is installed but not running, the script attempts to start it automatically and waits for the engine to become ready.
3. Run the server with [scripts/run-server-local-observability.ps1](../../../scripts/run-server-local-observability.ps1). The script forces `PORT=3000` and `BASE_URL=http://localhost:3000` so the local stack scrapes and links against the same address.
4. Open the dashboard at `http://localhost:3002/d/athena-auth-local-observability/athena-auth-local-observability`.

If Docker is unavailable but you still need local trace lookup, run the server with a captured JSON log file:

```powershell
powershell -File .\scripts\run-server-local-observability.ps1 -LogPath .\.local\athena-auth-observability.ndjson
```

## Find a trace from an auth error

When the API returns a payload like:

```json
{
  "message": "No handler found for this request",
  "version": "1.10.0",
  "traceId": "65bb3672-bb63-4038-9511-740a89620d25"
}
```

use that `traceId` directly in Grafana Explore:

1. Run [scripts/find-local-trace.ps1](../../../scripts/find-local-trace.ps1) with `-TraceId 65bb3672-bb63-4038-9511-740a89620d25`.
2. If you captured logs to a file instead of Loki, add `-LogPath .\.local\athena-auth-observability.ndjson`.
3. If you want the Grafana path instead, open `http://localhost:3002/explore`.
4. Pick the `Loki` datasource.
5. Start with `{service="athena-auth"}`.
6. Narrow it with `|= "65bb3672-bb63-4038-9511-740a89620d25"`.

Because the server now emits one wide completion event per HTTP request, the matching log line includes the request metadata that usually matters for auth incidents:

- `request_id` and `trace_id`
- `method`, `route`, `path`, and `matched_path`
- `tenant`
- `status`, `status_class`, and `outcome`
- `duration_ms`
- `origin`, `referer`, and `user_agent`
- `host`, `query_present`, and `disable_cookie_cache`
- `authorization_header_present`
- `cookie_header_present`
- `session_cookie_name`
- `session_cookie_present`
- `set_cookie`
- `metrics_enabled` and `loki_enabled`

## Other useful URLs

- Server metrics: `http://localhost:3000/metrics`
- Server metrics dashboard: `http://localhost:3000/metrics/view`
- Prometheus UI: `http://localhost:9090`
- Grafana UI: `http://localhost:3002`

## Shut it down

Run [scripts/stop-local-observability.ps1](../../../scripts/stop-local-observability.ps1).
