The llama.cpp server, llama-server, exposes a local HTTP API that is mostly OpenAI-compatible: chat requests go to /v1/chat/completions, embeddings to /v1/embeddings, the model list to /v1/models, and readiness to /health, all on port 9931 by default. That makes it a drop-in target for existing OpenAI SDK code, so the endpoints are the easy part. The hard part is everything around them: an always-on GPU, capacity you own, and model files and upgrades you maintain yourself. This guide lists the endpoints, shows a runnable call against the server, and prices self-hosting against a managed Llama API so you can see which side of that trade you are on.
What endpoints does the llama.cpp server expose?
llama-server serves two families of routes: native llama.cpp routes such as /completion, and OpenAI-compatible routes under /v1. The table below is the core set you will actually call.
| Path | Method | Purpose |
|---|---|---|
/v1/chat/completions | POST | OpenAI-compatible chat completions; supports streaming and JSON-schema output |
/v1/completions | POST | OpenAI-compatible text completions from a prompt |
/completion | POST | Native llama.cpp completion; expects a raw prompt, not a messages array |
/v1/embeddings | POST | OpenAI-compatible embeddings for text input |
/v1/models | GET | Returns the loaded model and its metadata |
/health | GET | Readiness check; {"status":"ok"} once the model is loaded, 503 while it is still loading |
/v1/messages | POST | Anthropic-compatible Messages API |
/props | GET | Server and model properties, including default generation settings |
/slots | GET | Per-slot state such as throughput, processed tokens, and sampling parameters |
/metrics | GET | Prometheus-format metrics; available only when the server runs with --metrics |
Endpoint set from the llama.cpp server reference, checked October 16, 2026.
Two details decide whether a given client talks to this server without changes. /v1/chat/completions is the route OpenAI SDK clients reach, while /completion is the llama.cpp-native equivalent and takes a prompt field instead of messages. And /health is public, so it needs no key and is the safest readiness probe for a container orchestration check.
How do you start llama-server and reach those endpoints?
You need two things before the server answers a request: a GGUF model file, and the llama-server binary built from the llama.cpp source. Clone and build it from the project root:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -t llama-serverThe binary lands at ./build/bin/llama-server. Start it with a model path and the flags that matter for serving:
./build/bin/llama-server \
-m ./models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
--host 0.0.0.0 \
--port 9931 \
-c 8192 \
-ngl 99 \
-np 4By default the server listens on 127.0.0.1:9931, so --host 0.0.0.0 is what exposes it beyond localhost. -ngl 99 offloads the maximum number of layers to the GPU, -c sets the context size, and -np sets the number of parallel slots, which is how many requests the single process can work on at once. None of the endpoints require authentication out of the box; add --api-key to make clients send Authorization: Bearer <key>, or leave it off and keep the server on a private network.
If you would rather not build it, the official image runs the same server:
docker run -p 9931:9931 -v /path/to/models:/models --gpus all \
ghcr.io/ggml-org/llama.cpp:server-cuda \
-m /models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
--host 0.0.0.0 --port 9931 -ngl 99A minimal call against /v1/chat/completions
The OpenAI-compatible route is the one worth testing first, because if it works, most of your existing tooling works too. From the shell:
curl http://localhost:9931/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "local-model",
"messages": [
{"role": "user", "content": "Explain llama.cpp API endpoints in one sentence."}
],
"max_tokens": 128
}'The model value is the server’s model alias and is ignored by a single-model server, but keep it present so the same request body also works in router mode. The OpenAI Python SDK talks to the same route with two changes — the base URL and a placeholder key, since the server ignores the key unless --api-key is set:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:9931/v1",
api_key="sk-no-key-required",
)
response = client.chat.completions.create(
model="local-model",
messages=[{"role": "user", "content": "Explain llama.cpp API endpoints in one sentence."}],
max_tokens=128,
)
print(response.choices[0].message.content)This is the same shape you would use against a hosted provider. Moving that call from a local server to Novita AI’s OpenAI-compatible API changes the base URL and the model name and leaves the request and response handling intact, which is why the endpoints are the least risky part of the self-hosting decision — the same compatibility contract is covered in more detail in What Is an OpenAI-Compatible API?.
Self-hosting cost vs a managed Llama API
Endpoints are portable; the bill is not. A self-hosted server is billed by the hour whether or not requests arrive, while a managed API is billed per token. The table compares the two on the factors that usually decide the choice.
| Cost factor | Self-hosted llama.cpp | Managed Llama API on Novita AI |
|---|---|---|
| What you pay for | GPU hours, rented or owned, plus engineering time | Input and output tokens |
| Billing while idle | You keep paying the hourly rate | Nothing |
| Price anchor | Novita GPU Cloud lists H100 SXM at $1.70 per GPU-hour and B200 SXM at $4.77 per GPU-hour | Llama 3.3 70B Instruct at $0.135 per million input tokens and $0.40 per million output tokens |
| Capacity and scaling | Fixed to the GPUs you run; you add more to grow | Elastic; the provider adds capacity |
| Model updates, quantization, tuning | Yours to run and re-verify | Provider-side |
| Data path | Requests stay on your hardware | Requests leave your network |
| Context window | Whatever the GGUF model and -c allow | As listed on the model page, for example 12,288 tokens for Llama 3.3 70B Instruct |
Token prices from the Novita AI model page and model list API, checked October 16, 2026; GPU rates from the Novita AI GPU Cloud page, checked October 16, 2026.
The gap is easiest to see on a worked example. Take a small application that processes 50 million input tokens and 10 million output tokens a month on Llama 3.3 70B Instruct. On the managed endpoint that is 50 × $0.135 + 10 × $0.40 = $6.75 + $4.00, or $10.75 a month. Run the same model yourself on one GPU that stays up, at the published H100 SXM rate of $1.70 per GPU-hour, and the month costs 730 hours × $1.70 ≈ $1,241, with or without traffic. The managed bill is about 1% of the always-on GPU bill, so self-hosting only closes that gap when you push roughly two orders of magnitude more traffic through the server — and only if one GPU can serve it.
That arithmetic is why the decision is really a utilization question, not a feature question. Self-hosting wins when the GPU runs hot enough that the flat hourly cost beats the per-token cost, which usually means a steady production workload rather than a bursty or early-stage one. Current GPU rates live on the Novita AI GPU Cloud page, and the model side is on the model catalog.
When to self-host llama.cpp, and when to use a managed Llama API
The endpoint compatibility means both sides run the same client code, so the choice comes down to control versus operational load.
Self-host llama.cpp when the data cannot leave your network, when you already operate GPUs and the marginal cost of another model is close to zero, or when you need a specific quantization, LoRA adapter, or chat template that no hosted provider exposes. A team running an existing GPU fleet for other work has a genuinely different cost basis than one renting a GPU purely to serve a Llama model.
Use a managed Llama API when traffic is bursty or unproven, when you want to skip capacity planning, or when you need a large context window without buying the VRAM that holds it. Novita AI serves Llama 3.1 8B Instruct at $0.02 per million input tokens and $0.05 per million output tokens, and Llama 4 Scout with a 131,072-token context, so the managed side scales from a cheap small model to a very long context without a hardware decision. The same catalog also carries Llama 3.3 70B Instruct as the mid-size default.
Two limitations of the self-hosted path are worth stating plainly, because they rarely appear in setup guides. First, llama-server is a single process on a single machine: concurrency is bounded by its -np slots and continuous batching, and it does not autoscale, so a traffic spike becomes a queue rather than an automatic scale-up. Second, you own the operational surface — model downloads and quantization choices, upgrades, TLS and a reverse proxy in front of the port, authentication, and monitoring through /slots, /props, and /metrics — and a crashed process is your outage, with no provider SLA behind it. The break-even math from the previous section assumes the server is up and serving; keeping it that way is the work you are taking on.
Conclusion
The llama.cpp API endpoints are the portable, low-risk part of self-hosting: /v1/chat/completions, /v1/embeddings, /v1/models, and /health speak the same OpenAI-compatible contract your existing code already uses, and the server runs on a single command. The decision rests on the hour-by-hour GPU bill and the operational load, not on the API surface. Below very high sustained utilization, a managed Llama API is cheaper because you pay only for the tokens you send and receive; self-hosting earns its keep when privacy, an existing GPU fleet, or a custom model forces the choice. If you want the endpoints without the GPU, start from the Llama models on Novita AI and point the same SDK code at the base URL.
FAQ
Is the llama.cpp server OpenAI-compatible?
Mostly. It exposes /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models under an OpenAI-style contract, so OpenAI SDK clients connect after changing the base URL and the model name. Features outside basic chat, such as tool calling and structured output, depend on the loaded model’s chat template, so verify the specific model before relying on them.
What port does llama-server use by default?
Port 9931 on 127.0.0.1. Start the server with --host 0.0.0.0 to accept connections from other machines, and --port to move it. The official Docker image examples publish port 8080, so match the container’s internal port to whatever you passed to --port.
Do I need an API key to call the llama.cpp server?
No, not by default. The server accepts requests without authentication unless you start it with --api-key, which then requires Authorization: Bearer <key> on each call. Leaving it open is only reasonable when the server is bound to a private interface; anything reachable from a network needs the key and, ideally, a reverse proxy in front of it.
Can I use the OpenAI Python SDK against llama.cpp?
Yes. Set base_url to http://localhost:9931/v1 and supply any placeholder api_key, because the SDK requires the field but the server ignores it unless --api-key is set. The request and response objects keep the OpenAI shape, so response.choices[0].message.content returns the generated text.
How many concurrent requests can llama-server handle?
As many as its parallel slots, set with -np, and no more. The server handles slots concurrently through continuous batching, but it does not add capacity on demand, so the practical ceiling is the GPU and the slot count you configured. For elastic concurrency you need a managed endpoint or an orchestration layer that starts more server processes.
When is self-hosting llama.cpp cheaper than a managed Llama API?
When the GPU runs hot enough that its flat hourly cost beats the per-token bill for the same traffic. A single always-on H100-class GPU costs on the order of a thousand dollars a month, while a modest workload on a managed Llama model can cost single-digit dollars a month, so self-hosting usually needs a steady, high-volume production load — or an existing GPU you are already paying for — before it wins.
Sources
- llama.cpp server reference — endpoint list, default port, build and startup flags; checked October 16, 2026.
- Novita AI model list API — Llama model IDs, prices, and context windows; checked October 16, 2026.
- Novita AI Llama 3.3 70B Instruct model page — pricing and context window; checked October 16, 2026.
- Novita AI GPU Cloud — H100 SXM and B200 SXM per-GPU-hour rates; checked October 16, 2026.
