# Understanding the Tokio Runtime

This document explains how the Tokio async runtime works internally. Use this knowledge to reason about trace data, diagnose performance problems from first principles, and recommend code changes.

## The execution model

Tokio runs async tasks on a thread pool of **worker threads**. Each worker runs a loop:

1. Pick a task from its local queue (or steal from another worker, or take from the global injection queue)
2. Call `.poll()` on the task's future
3. If the future returns `Poll::Pending`, the task is suspended until something wakes it
4. If the future returns `Poll::Ready`, the task is complete
5. Repeat

A **poll** is the fundamental unit of work. During a single poll, the future runs synchronously on the worker thread until it hits an `.await` point that returns `Pending`. Nothing else can run on that worker thread until the poll completes.

### current_thread vs multi_thread

- **`new_current_thread()`**: One worker thread. All tasks share a single thread. Simple, no synchronization overhead, but all tasks compete for one CPU. Common for clients, CLIs, and lightweight servers.
- **`new_multi_thread()`**: Multiple worker threads (default: one per CPU core). Tasks can run in parallel. Uses work-stealing to balance load.

On a current_thread runtime, if one task's poll takes 500µs, every other task is blocked for that entire duration. On a multi_thread runtime with N workers, up to N tasks can be polled simultaneously.

## Waking and scheduling

When a task `.await`s something (a socket read, a timer, a channel receive), it returns `Pending` and registers a **waker**. When the awaited resource becomes ready (data arrives on the socket, timer fires, channel has a message), the waker is called, which places the task back on a worker's run queue.

The **wake-to-poll delay** is the time between `Waker::wake()` being called and the task actually being polled. This delay has two components:

1. **Queue wait**: The worker is busy polling other tasks. The woken task sits in the queue until the worker finishes its current poll and gets to it.
2. **Kernel scheduling wait**: The worker thread itself was parked (sleeping) and the OS needs to reschedule it onto a CPU core.

High wake-to-poll delays are the primary cause of tail latency in async applications. They mean "the runtime knew this task had work to do, but couldn't get to it fast enough."

## Cooperative scheduling and yield points

Tokio uses **cooperative scheduling**: a task runs until it voluntarily yields (by hitting a `Pending` `.await`). There is no preemption. If a task does CPU-heavy work or processes many items in a loop without awaiting, it monopolizes the worker.

Tokio has a built-in **coop budget** (currently 128 operations). After a task has done 128 "coop-aware" operations (socket reads, channel receives, etc.) in a single poll, Tokio forces it to yield by making the next operation return `Pending` even if data is available. This prevents a single busy task from starving others indefinitely.

However, the coop budget only applies to Tokio-aware operations. A tight loop doing synchronous work, or calling non-Tokio I/O, will not trigger a yield. In these cases, you need explicit `tokio::task::yield_now().await` calls.

### When to use yield_now()

Insert `yield_now().await` when a task processes multiple items in a loop and each iteration is fast but the total loop time could be long:

- Processing a batch of messages from a channel
- Handling pipelined requests on a connection (e.g., Redis pipelining)
- Iterating over a large data structure with periodic async work

The tradeoff: yielding adds overhead per iteration (the task must be rescheduled), which increases p50 latency slightly. But it dramatically reduces p99 latency under concurrency because other tasks get a chance to run between iterations.

**Rule of thumb**: If a single poll handles N items and each takes T microseconds, the worst-case delay for other tasks is N×T. If N×T exceeds your latency budget, add yields.

## How poll duration affects tail latency

On a single-threaded runtime with C concurrent connections, the worst-case wake-to-poll delay is approximately:

```
worst_case_delay ≈ (C - 1) × avg_poll_duration
```

On a multi-threaded runtime with W workers and C connections:

```
worst_case_delay ≈ ceil(C / W) × avg_poll_duration
```

This is why p99 latency often scales linearly with connection count when poll durations are not controlled. The fix is always the same: make individual polls shorter, either by yielding, by moving work off the runtime, or by reducing per-poll work.

### Worked example

A Redis-like server on `current_thread` with 4 connections. Each poll processes a batch of pipelined commands, taking ~225µs. When connection A's task is polled, connections B, C, and D wait. Worst case: D waits for A + B + C = 675µs. Add kernel scheduling overhead and you get p99 latencies approaching 1ms+.

Fix: `yield_now().await` after each command. Now each poll handles one command (~5-10µs of actual work + I/O await), and the maximum wait drops to ~30-40µs.

## What makes a poll long?

A poll is "long" when the future does significant work between await points. Common causes:

1. **Synchronous I/O on the runtime**: File reads, DNS resolution, blocking HTTP clients. These should use `spawn_blocking()`.
2. **CPU-intensive computation**: Serialization, compression, cryptography, large collection operations. Move to `spawn_blocking()` or break up with `yield_now()`.
3. **Lock contention**: Holding a `std::sync::Mutex` across an await, or waiting on a contended lock. Use `tokio::sync::Mutex` or restructure to avoid holding locks during polls.
4. **Batch processing without yielding**: Processing all available items in a loop (pipelined requests, channel drains, batch inserts) without giving other tasks a chance to run.
5. **Memory allocation**: Large allocations or heavy allocator contention can add microseconds to hundreds of microseconds.

In traces, look at `poll.cpuSamples` (what was on-CPU during the poll) and `poll.schedSamples` (what caused the OS to deschedule the worker — indicates blocking syscalls).

## The global injection queue

When a task is woken and the target worker's local queue is full, or when the wake comes from outside the runtime (e.g., a `spawn_blocking` completion), the task goes into the **global injection queue**. Workers periodically check this queue, but it's slower than the local queue due to synchronization.

High global queue depth (visible in `QueueSample` events) means:
- Workers can't keep up with incoming work
- Too many tasks are being woken from outside the runtime
- Workers are spending too long on individual polls

## Worker parking and unparking

When a worker has no tasks to run, it **parks** (goes to sleep via an OS futex/condvar). When new work arrives, it's **unparked** (woken by the OS). The trace captures:

- **Park → Unpark duration**: How long the worker slept. Long parks mean the worker had nothing to do.
- **CPU time on active spans**: The `ratio` field (CPU time / wall time) shows whether the worker was actually on-CPU while active. A ratio < 1.0 means the kernel descheduled the worker — it was runnable but another process/thread got the CPU.
- **schedWait on Unpark**: How long the kernel took to actually schedule the worker thread after it was woken. High values indicate CPU contention at the OS level.

## Connecting trace data to application behavior

When analyzing a trace, think in terms of this causal chain:

1. **External event** (network data arrives, timer fires) → **wake** is sent to a task
2. Task enters a worker's **run queue** → wake-to-poll delay starts
3. Worker finishes its current poll → picks up the woken task → **poll starts**
4. Future runs: reads data, processes command, writes response → **poll ends**
5. If more work is available, the task may be polled again immediately; otherwise it awaits and goes back to step 1

Latency problems map to specific parts of this chain:
- **High wake-to-poll delay**: Workers are busy (step 2-3). Fix: shorter polls, more workers, or yield points.
- **Long poll duration**: The task itself is slow (step 4). Fix: move blocking work off-runtime, add yields, optimize hot paths.
- **High kernel sched wait**: OS can't schedule workers fast enough (step 3). Fix: reduce CPU contention, check for noisy neighbors, pin workers to cores.
- **High queue depth**: More work arriving than workers can handle. Fix: add backpressure, increase worker count, optimize poll duration.

## Common fixes and their tradeoffs

| Problem | Fix | Tradeoff |
|---------|-----|----------|
| Long polls from batch processing | `yield_now().await` in the loop | Slightly higher p50 from yield overhead |
| Blocking I/O on runtime | `spawn_blocking()` | Thread pool overhead, data must be `Send` |
| CPU-heavy computation | `spawn_blocking()` or `rayon` | Same as above |
| High wake-to-poll delay, single thread | Switch to `new_multi_thread()` | More memory, synchronization overhead |
| High wake-to-poll delay, multi thread | Increase worker count or reduce poll duration | More CPU usage when idle |
| Lock contention | `tokio::sync::Mutex`, sharding, or lock-free structures | Complexity |
| Task leak (unbounded spawning) | Bounded channels, semaphores, `JoinSet` | Backpressure may slow producers |
