---
phase: 22-course-load-testing
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - pmcp-course/src/part7-observability/ch18-03-performance.md
autonomous: true
requirements:
  - CRLT-01
  - CRLT-02
  - CRLT-03

must_haves:
  truths:
    - "Learner can follow step-by-step instructions to run their first load test using cargo pmcp loadtest"
    - "Learner can author a TOML config file from scratch after reading the guided walkthrough"
    - "Learner understands loadtest init schema discovery and can use it against a running server"
    - "Learner can configure staged load profiles (ramp-up/hold/ramp-down) for capacity planning"
    - "Learner can interpret HdrHistogram percentiles (P50/P95/P99) and coordinated omission correction"
    - "Learner can load test a deployed MCP server and derive capacity planning conclusions from the results"
  artifacts:
    - path: "pmcp-course/src/part7-observability/ch18-03-performance.md"
      provides: "Complete hands-on load testing tutorial replacing stub"
      min_lines: 500
      contains: "cargo pmcp loadtest"
  key_links:
    - from: "pmcp-course/src/part7-observability/ch18-03-performance.md"
      to: "pmcp-book/src/ch14-performance.md"
      via: "Cross-reference to book chapter for full reference documentation"
      pattern: "ch14-performance"
---

<objective>
Write the complete Ch 18-03: Performance Optimization hands-on tutorial for pmcp-course, replacing the current one-line stub.

Purpose: Course learners need a step-by-step, teaching-oriented tutorial that walks them through load testing an MCP server using `cargo pmcp loadtest`. Unlike the book's Ch 14 (which is a reference), this chapter teaches by doing -- each concept is introduced through a hands-on exercise the learner performs, with explanations woven around the practical work.

Output: `pmcp-course/src/part7-observability/ch18-03-performance.md` fully written as a hands-on tutorial.
</objective>

<execution_context>
@./.claude/get-shit-done/workflows/execute-plan.md
@./.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md

The existing file is a one-line stub: `# Performance Optimization`

**CRITICAL STYLE GUIDANCE:** This is a COURSE chapter, not a book chapter. Read these files to match the course's teaching style:
- `pmcp-course/src/part1-foundations/ch02-03-calculator.md` — exemplar hands-on tutorial style
- `pmcp-course/src/part4-testing/ch12-remote-testing.md` — testing chapter teaching style
- `pmcp-course/src/part4-testing/ch12-01-remote.md` — sub-chapter detail level

The course style features:
- "Learning Objectives" at the top (bullet list of what learner will be able to do)
- Step-by-step instructions with numbered sequences the learner follows
- ASCII diagrams for architecture/flow visualization
- Tables for quick reference (key-value summaries)
- "Practice Ideas" informal exercises at the bottom
- Teaching voice: "Let's...", "You'll...", "Try this...", second person throughout
- Code blocks with inline comments explaining each line
- "Why this matters" explanations after each concept

**Source material:** The book's Ch 14 (`pmcp-book/src/ch14-performance.md`) contains all the reference content. Read it during execution to extract accurate details. The course chapter should TEACH the same material hands-on, not duplicate the reference prose.

**Source code files** (READ during execution for accuracy):
- `cargo-pmcp/src/commands/loadtest/mod.rs` — CLI structure
- `cargo-pmcp/src/commands/loadtest/run.rs` — Run command
- `cargo-pmcp/src/commands/loadtest/init.rs` — Init command, schema discovery
- `cargo-pmcp/src/loadtest/config.rs` — TOML config schema
- `cargo-pmcp/src/loadtest/engine.rs` — Execution modes
- `cargo-pmcp/src/loadtest/metrics.rs` — HdrHistogram, coordinated omission
- `cargo-pmcp/src/loadtest/breaking.rs` — Breaking point detection
- `cargo-pmcp/src/loadtest/report.rs` — JSON report schema
- `cargo-pmcp/src/loadtest/summary.rs` — Terminal output format
</context>

<tasks>

<task type="auto">
  <name>Task 1: Write Ch 18-03 tutorial sections 1-5 — Setup through Staged Load Profiles</name>
  <files>pmcp-course/src/part7-observability/ch18-03-performance.md</files>
  <action>
Read the book's Ch 14 (`pmcp-book/src/ch14-performance.md`) for reference content accuracy, and read these course chapters for style reference:
- `pmcp-course/src/part1-foundations/ch02-03-calculator.md` (hands-on tutorial exemplar)
- `pmcp-course/src/part4-testing/ch12-remote-testing.md` (testing chapter structure)
- `pmcp-course/src/part4-testing/ch12-01-remote.md` (sub-chapter detail level)

Also read source code for accuracy:
- `cargo-pmcp/src/loadtest/config.rs` for TOML config schema
- `cargo-pmcp/src/commands/loadtest/init.rs` for schema discovery behavior
- `cargo-pmcp/src/loadtest/engine.rs` for flat vs staged execution

Write `pmcp-course/src/part7-observability/ch18-03-performance.md` with these sections in course teaching style:

**1. Title and Learning Objectives** (# Performance Optimization)
- Brief paragraph: "In this chapter, you'll learn to load test your MCP servers..."
- ## Learning Objectives (bullet list):
  - Run a load test against an MCP server using `cargo pmcp loadtest`
  - Write TOML configuration files defining test scenarios
  - Use schema discovery to auto-generate configs from running servers
  - Configure staged load profiles for capacity planning
  - Interpret HdrHistogram percentiles and understand coordinated omission
  - Find your server's breaking point under load
  - Plan capacity for production deployments

**2. Why Load Test MCP Servers?** (## Why Load Test MCP Servers?)
- Teaching introduction: Why this matters for MCP specifically (many servers, few clients)
- ASCII diagram: Testing Pyramid showing load testing at the top
- Brief comparison: "Unlike HTTP load testing tools, `cargo pmcp loadtest` understands MCP protocol semantics"
- Cross-reference: "For the complete reference documentation, see [Chapter 14 of the PMCP Book](../../pmcp-book/src/ch14-performance.md). This tutorial teaches the same material hands-on."

**3. Your First Load Test** (## Your First Load Test)
- Teaching framing: "Let's run your first load test in three steps"
- ### Step 1: Start Your Server
  - Use the calculator server from Ch 2 as the test target
  - Show the command to start it: `cargo run --release --bin calculator`
  - "Make sure it's running on http://localhost:3000/mcp"
- ### Step 2: Generate a Config
  - Walk through `cargo pmcp loadtest init http://localhost:3000/mcp`
  - Explain what schema discovery does (connects, discovers tools/resources/prompts, generates TOML)
  - Show the generated `.pmcp/loadtest.toml` output
  - "Notice how it discovered your calculator's tools automatically"
- ### Step 3: Run the Test
  - Walk through `cargo pmcp loadtest run http://localhost:3000/mcp`
  - Show the expected k6-style terminal output (ASCII art header + metrics)
  - Explain each metric line briefly: "p50=12ms means half your requests completed in under 12ms"
  - "Congratulations — you've run your first MCP load test!"

**4. Understanding the Config File** (## Understanding the Config File)
- Teaching framing: "Now let's understand what that generated config actually means"
- ### The [settings] Block
  - Walk through each field with teaching explanation:
    - `virtual_users`: "Think of these as simulated clients hammering your server simultaneously"
    - `duration_secs`: "How long the test runs"
    - `timeout_ms`: "When to give up on a single request"
    - `expected_interval_ms`: "We'll explain this when we cover coordinated omission correction"
- ### The [[scenario]] Blocks
  - Explain weight-based proportional scheduling with a concrete example
  - Walk through each operation type: tools/call, resources/read, prompts/get
  - "Try this:" exercise — modify the config to change weights and rerun
- ### Writing Your Own Config from Scratch
  - Step-by-step guide: create a new TOML file manually
  - Show a complete annotated config the learner types out
  - Explain validation rules (at least one scenario, weights > 0)

**5. Staged Load Profiles** (## Staged Load Profiles)
- Teaching framing: "Flat load tests a constant level. But what if you want to find your server's limits?"
- ### Flat Load vs Staged Load
  - ASCII diagram showing flat load (constant VUs)
  - ASCII diagram showing staged load (ramp-up, hold, ramp-down)
  - When to use each (brief teaching comparison)
- ### Building Your First Staged Profile
  - Step-by-step: add [[stage]] blocks to the config
  - Walk through ramp-up -> hold -> ramp-down pattern
  - "When stages are present, `settings.virtual_users` is ignored — the stages control everything"
  - "Try this:" exercise — create a 3-stage profile and run it
  - Show the terminal output with `[stage 2/3]` indicator
- ### Designing Profiles for Different Goals
  - Brief table: Goal -> Profile Shape (capacity planning, soak testing, stress testing)
  - Teaching summary of when to use each

Leave a clear `<!-- CONTINUED IN TASK 2 -->` comment at the end.

IMPORTANT: Use teaching voice throughout — "Let's...", "You'll see...", "Try this...", "Notice how...". Every concept is introduced through something the learner DOES, not just reads about. Include ASCII diagrams for visual concepts. Match the course style from the reference chapters, not the book style.
  </action>
  <verify>
    <automated>grep -c "## " pmcp-course/src/part7-observability/ch18-03-performance.md | xargs test 4 -le</automated>
    Verify the file has at least 4 major sections (## headers), contains "Learning Objectives", "cargo pmcp loadtest", "loadtest init", "loadtest run", "[settings]", "[[scenario]]", "[[stage]]", "Try this".
    Run: grep -cE "(Learning Objectives|cargo pmcp loadtest|Try this|\\[settings\\]|\\[\\[scenario\\]\\]|\\[\\[stage\\]\\])" pmcp-course/src/part7-observability/ch18-03-performance.md
    Should return 8+.
  </verify>
  <done>
    Ch 18-03 has complete tutorial sections 1-5 covering: learning objectives, motivation, first load test walkthrough, config file deep dive, and staged load profiles. All in course teaching style with hands-on steps, ASCII diagrams, and "Try this" exercises. Accurate to source code.
  </done>
</task>

<task type="auto">
  <name>Task 2: Write Ch 18-03 tutorial sections 6-10 — Metrics, Breaking Points, Deployed Server Example, and Capacity Planning</name>
  <files>pmcp-course/src/part7-observability/ch18-03-performance.md</files>
  <action>
Read the book's Ch 14 (`pmcp-book/src/ch14-performance.md`) sections on metrics, breaking point, and JSON reports for reference content accuracy.

Also read source code for accuracy:
- `cargo-pmcp/src/loadtest/metrics.rs` for HdrHistogram, coordinated omission
- `cargo-pmcp/src/loadtest/breaking.rs` for breaking point detection algorithm
- `cargo-pmcp/src/loadtest/report.rs` for JSON report schema
- `cargo-pmcp/src/loadtest/summary.rs` for terminal output metric names

Remove the `<!-- CONTINUED IN TASK 2 -->` comment and append these sections:

**6. Reading Your Results** (## Reading Your Results)
- Teaching framing: "You've run tests — now let's understand what the numbers mean"
- ### Percentiles: P50, P95, P99
  - Teach with analogy: "P50 is 'typical', P95 is 'almost worst case', P99 is 'the unlucky 1%'"
  - Table showing what each percentile means for a user
  - "Why averages lie" teaching moment with example
- ### The Terminal Output Explained
  - Show full k6-style output and annotate each metric line
  - `mcp_req_duration`, `mcp_req_success_count`, `mcp_req_error_count`, `mcp_req_error_rate`, `mcp_req_throughput`, `mcp_req_total`, `mcp_req_elapsed`
  - Error classification breakdown (timeout, jsonrpc, http, connection)
  - Per-tool metrics table
- ### HdrHistogram and Coordinated Omission
  - Teach the problem: "When your server stalls, naive tools stop measuring — and miss the worst latency"
  - Explain `expected_interval_ms` and how correction fills synthetic samples
  - Teaching guideline: match to your server's expected response time
  - "This is why our numbers are more accurate than simple averages or basic histograms"

**7. Finding Your Server's Breaking Point** (## Finding Your Server's Breaking Point)
- Teaching framing: "Every server has a limit. Let's find yours."
- ### What the Breaking Point Detector Does
  - Teach the rolling window concept simply: "It watches your server's behavior over time and flags when things start going wrong"
  - Two detection conditions explained in learner terms:
    1. Error rate suddenly spikes (>10% and >2x baseline)
    2. Latency suddenly jumps (P99 >3x baseline)
  - "It's self-calibrating — it learns what 'normal' looks like for YOUR server"
- ### Hands-On: Find Your Calculator Server's Breaking Point
  - Step-by-step staged config that ramps from 10 to 100 VUs
  - Walk learner through running it
  - "Watch for the breaking point warning in the terminal"
  - Show expected output: `Breaking point detected at NN VUs...`
  - "Your number will be different — it depends on your machine's resources"
- ### Reading the JSON Report
  - Show where reports are saved: `.pmcp/reports/`
  - Walk through the JSON structure with teaching annotations
  - "Try this:" use `jq` to extract the breaking point VU count

**8. Load Testing a Deployed Server** (## Load Testing a Deployed Server)
- Teaching framing: "Local testing tells you the floor, not the ceiling. Let's test against a real deployment."
- ### Testing Against a Remote Server
  - Walk through pointing loadtest at a deployed URL
  - Discuss considerations: network latency, cold starts, auth headers
  - Config adjustments for remote testing (higher timeouts, realistic VU counts)
- ### A Practical Example: Staging Environment Load Test
  - Complete walkthrough: generate config from staging server, customize for realistic traffic, run the test
  - Show how to use `--vus` and `--duration` CLI overrides for quick tests
  - Interpret the results: "If P99 is under 500ms at 20 VUs, your staging can handle ~20 concurrent clients"
- ### Capacity Planning from Results
  - Teaching framework: "Here's how to turn load test numbers into deployment decisions"
  - Table: VU Count -> P99 -> Error Rate -> Conclusion
  - Example: "If breaking point is 50 VUs and you expect 30 peak concurrent users, you have 40% headroom"
  - Guidance on when to scale up vs optimize
  - "Rule of thumb: plan for 2x your expected peak"

**9. Quick Reference** (## Quick Reference)
- CLI commands table (run, init with common flags)
- Config file cheat sheet (settings, scenario types, stage blocks)
- Metric names reference table
- "For the complete reference, see [Chapter 14 of the PMCP Book](../../pmcp-book/src/ch14-performance.md)"

**10. Chapter Summary and Practice Ideas** (## Chapter Summary)
- Key takeaways as bullet list
- ## Practice Ideas
  - 4-5 informal exercises:
    1. Load test the calculator with different VU counts and graph P99 vs VUs
    2. Create a staged profile that simulates morning ramp-up traffic
    3. Find the breaking point of a database-backed MCP server (from Ch 3)
    4. Load test a deployed server (from Ch 8) and compare local vs remote results
    5. Write a shell script that fails if P99 exceeds a threshold (CI/CD preview)
- Navigation link: *Return to [Operations and Monitoring](./ch18-operations.md)*

IMPORTANT: Maintain teaching voice throughout. Every section starts with what the learner DOES, then explains. The deployed server example (section 8) is a complete practical walkthrough, not just reference material — the learner should feel they could follow it against a real server. Include ASCII diagrams where they aid understanding.
  </action>
  <verify>
    <automated>wc -l pmcp-course/src/part7-observability/ch18-03-performance.md | awk '{if ($1 >= 500) print "PASS: "$1" lines"; else print "FAIL: only "$1" lines, need 500+"; exit ($1 < 500)}'</automated>
    Also verify key content:
    Run: grep -cE "(HdrHistogram|breaking.point|coordinated omission|Capacity Planning|Practice Ideas|mcp_req_|deployed|staging)" pmcp-course/src/part7-observability/ch18-03-performance.md
    Should return 10+ matches confirming all major topics are covered.
  </verify>
  <done>
    Ch 18-03 is complete with: metrics interpretation (HdrHistogram, percentiles, coordinated omission), breaking point detection hands-on, deployed server load testing practical example, capacity planning guidance, quick reference, and practice ideas. All in course teaching style with hands-on steps. Covers CRLT-01, CRLT-02, and CRLT-03 fully.
  </done>
</task>

</tasks>

<verification>
1. `pmcp-course/src/part7-observability/ch18-03-performance.md` exists and is > 500 lines
2. Chapter covers all assigned requirements:
   - CRLT-01: Hands-on tutorial using `cargo pmcp loadtest` with step-by-step instructions
   - CRLT-02: TOML config authoring, `loadtest init` schema discovery, staged load profiles, result interpretation
   - CRLT-03: Practical deployed server load testing example with capacity planning guidance
3. Teaching style matches existing course chapters (learning objectives, step-by-step, ASCII diagrams, "Try this" exercises, practice ideas)
4. Cross-references book Ch 14 for complete reference documentation
5. All CLI flags, config fields, metric names, and threshold values are accurate to source code
</verification>

<success_criteria>
- A course learner can follow the tutorial start-to-finish and successfully load test an MCP server
- The chapter teaches through doing, not just reading — every concept is introduced via a hands-on step
- Writing style matches the established course voice (second person, teaching-oriented, "Let's...", "Try this...")
- Deployed server example is practical enough that a learner could follow it against a real staging environment
- Capacity planning section gives actionable guidance, not just theory
</success_criteria>

<output>
After completion, create `.planning/phases/22-course-load-testing/22-01-SUMMARY.md`
</output>
