llama.cpp integration

llama.cpp is a lightweight, CPU-first inference engine for LLMs using GGUF format models. RavenClaws supports llama.cpp via the generic openai-compatible provider — no special configuration needed.

Quick start

1. Start llama.cpp server

shell

# Download a GGUF model
wget https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/mistral-7b-instruct-v0.3.Q4_K_M.gguf

# Start the server
llama-server -m mistral-7b-instruct-v0.3.Q4_K_M.gguf --port 8080

Or using Docker:

shell

docker run --rm -p 8080:8080 \
  -v $(pwd)/models:/models \
  ghcr.io/ggerganov/llama.cpp:server \
  -m /models/mistral-7b-instruct-v0.3.Q4_K_M.gguf \
  --port 8080

2. Configure RavenClaws

Via environment variables:

shell

export RAVENCLAWS__LLM__PROVIDER="openai-compatible"
export RAVENCLAWS__LLM__ENDPOINT="http://localhost:8080/v1/chat/completions"
export RAVENCLAWS__LLM__MODEL="mistral-7b-instruct-v0.3"

ravenclaws --exec "What is the capital of France?"

Configuration reference

Field	Value	Description
`provider`	`openai-compatible`	Must be set to `openai-compatible`
`endpoint`	`http://localhost:8080/v1/chat/completions`	llama.cpp's OpenAI-compatible endpoint
`model`	(model name)	The GGUF model loaded in llama.cpp
`api_key`	(optional)	Not needed for local llama.cpp

Tool-calling support

Backend	Tool calling	Notes
llama.cpp	❌ None	GGUF format does not support structured tool calling
RavenClaws fallback	✅ Text-based parsing	Detects `TOOL_CALL:` / `ARGS:` patterns automatically

llama.cpp does not support structured function calling (OpenAI tools format). However, RavenClaws's agent loop includes a text-based fallback that detects TOOL_CALL: and ARGS: patterns in the model's response text. For best results, use --no-final-required:

shell

ravenclaws --provider openai-compatible \
  --endpoint http://localhost:8080/v1/chat/completions \
  --model mistral-7b-instruct-v0.3 \
  --exec "List the files in the current directory" \
  --no-final-required

Troubleshooting

Problem	Likely cause	Solution
Connection refused	llama.cpp not running	Start `llama-server` with your GGUF model
Model not found	Wrong model name	Check `curl http://localhost:8080/v1/models`
Empty response	Model not fully loaded	Wait for llama.cpp to finish loading
Slow inference	CPU-only inference	Use a smaller quantized model (Q4_K_M or Q3_K_S)
Tool calls not working	GGUF doesn't support tools	Use `--no-final-required` and rely on text-based fallback
High memory usage	Large model on CPU	Use a smaller GGUF quantization (Q4_K_M is a good balance)

Performance tips

Use quantized models — Q4_K_M offers the best quality-to-speed ratio
Match context window — Set --ctx-size to match your task needs (4096+ recommended for agent tasks)
Batch size — Increase --batch-size for faster prompt processing
GPU offloading — Use -ngl N to offload N layers to GPU if available

Multi-model with llama.cpp

You can use llama.cpp alongside other providers in multi-model mode:

ravenclaws.toml

[llm]
provider = "multi"

[[llm.models]]
provider = "openai-compatible"
endpoint = "http://localhost:8080/v1/chat/completions"
model = "mistral-7b-instruct-v0.3"

[[llm.models]]
provider = "openai"
model = "gpt-4o"
api_key = "${OPENAI_API_KEY}"