vLLM integration

vLLM is a high-throughput, GPU-accelerated inference engine. RavenClaws supports vLLM via the generic openai-compatible provider — no special configuration needed.

Quick start

1. Start vLLM server

Using Docker (recommended):

shell

docker run --rm --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --port 8000

Or using pip:

shell

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --port 8000

2. Configure RavenClaws

Via environment variables:

shell

export RAVENCLAWS__LLM__PROVIDER="openai-compatible"
export RAVENCLAWS__LLM__ENDPOINT="http://localhost:8000/v1/chat/completions"
export RAVENCLAWS__LLM__MODEL="mistralai/Mistral-7B-Instruct-v0.3"

ravenclaws --exec "What is the capital of France?"

Configuration reference

Field	Value	Description
`provider`	`openai-compatible`	Must be set to `openai-compatible`
`endpoint`	`http://localhost:8000/v1/chat/completions`	vLLM's OpenAI-compatible endpoint
`model`	(model name)	The model loaded in vLLM
`api_key`	(optional)	Not needed for local vLLM

Tool-calling support

Backend	Tool calling	Notes
vLLM (with tool support)	✅ Structured	Works with models that support OpenAI tool format
vLLM (no tool support)	⚠️ Text fallback	RavenClaws detects `TOOL_CALL:` / `ARGS:` patterns

vLLM supports structured function calling for models that include it in their tokenizer config. For models without native tool support, RavenClaws falls back to text-based parsing.

Troubleshooting

Problem	Likely cause	Solution
Connection refused	vLLM not running	Start vLLM with `--port 8000`
Model not found	Wrong model name	Check `curl http://localhost:8000/v1/models`
CUDA out of memory	Model too large for GPU	Use a smaller model or reduce `--max-model-len`
Slow first response	Model loading	Wait for vLLM to finish loading; subsequent requests are fast
Tool calls not working	Model lacks tool support	Use `--no-final-required` for text-based fallback

Multi-model with vLLM

You can use vLLM alongside other providers in multi-model mode:

ravenclaws.toml

[llm]
provider = "multi"

[[llm.models]]
provider = "openai-compatible"
endpoint = "http://localhost:8000/v1/chat/completions"
model = "mistralai/Mistral-7B-Instruct-v0.3"

[[llm.models]]
provider = "openai"
model = "gpt-4o"
api_key = "${OPENAI_API_KEY}"