llama.cpp vs Ollama: Which Local Runtime

Use Ollama when you want to pull a model and run it with one command, with updates and a model library handled for you. Use llama.cpp when you want direct control of the engine: build flags, backends, quantisation and server options. Both are MIT licensed and both serve an OpenAI-style API. Most people should start with Ollama and move down to llama.cpp only when a setting they need is out of reach. Checked 31 October 2026.

The short version

The rest of the page explains the gap between those rows. They are closer than they look, because one of them is an engine and the other is a product built around running engines.

What each one is

llama.cpp

The repository description is two words of promise: LLM inference in C/C++. It is the project that made it practical to run large models on ordinary hardware. The README lists its main traits:

• Apple silicon is described as a first-class citizen, optimised through ARM NEON, Accelerate and Metal.
• Custom CUDA kernels run models on NVIDIA GPUs, with AMD support through HIP and Moore Threads support through MUSA.
• Vulkan and SYCL backends are supported. The backend list in the README also names CANN, Hexagon, WebGPU and OpenVINO (marked in progress).
• Quantisation runs from 1.5-bit up to 8-bit integer, for faster inference and lower memory use.
• CPU plus GPU hybrid inference lets you partly accelerate a model that is larger than your video memory.

On 28 October 2026 the repository's latest release was tagged v0.8.0, and the licence on GitHub is MIT. It ships a command-line chat tool, an HTTP server, and a built-in web interface against that server. The README's quick start now uses short commands: llama cli to download and run a model, and llama serve for the API server.

Ollama

Ollama's repository describes itself as the way to get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. It is MIT licensed, and the latest release on 28 October 2026 was v0.40.13.

It is a product more than a library. You install an app for macOS, Windows or Linux, you pick a model from its library by name, and it runs a local HTTP server on port 11434. It handles downloading, loading, unloading and updating. Its docs also cover Docker, a Modelfile for importing and customising models, a context-length setting and hardware support notes. For a fuller view see our Ollama vs LM Studio page.

Side-by-side facts

Setup, side by side

llama.cpp

The README gives an install script for macOS and Linux and a PowerShell one for Windows:

curl -LsSf https://llama.app/install.sh | sh
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

You can also download pre-built binaries from the releases page, run it in Docker, or build from source with the project's build guide. Building from source is the route to a specific backend. You choose the flag for CUDA, Vulkan or another backend at build time, which is a strength if you need it and a chore if you do not.

The -hf flag means you can point at a Hugging Face repository and it downloads the GGUF file for you. That gives you the freshest model builds the day a lab publishes them, without waiting for a library entry.

Ollama

ollama run gemma3

That model name is an example. The command pulls the model on first use and opens a chat. The server is already running in the background for any other tool that wants to call it. The FAQ shows the knobs that matter: OLLAMA_HOST to listen on your network, OLLAMA_MODELS to move the model folder, and keep_alive to keep a model in memory or unload it.

Control versus convenience

This is the real decision, and it is worth spelling out.

What llama.cpp gives you that Ollama hides. You see the flags. You choose how many layers to put on the GPU, the context size, the batch size, and the quantisation file you download. The README's mention of CPU plus GPU hybrid inference is a good example: a model bigger than your VRAM can still run, with some layers on the CPU, and you tune the split. If you are squeezing a large model onto a small card, this control is worth the effort.

What Ollama gives you that llama.cpp leaves to you. Model naming, downloads, version tracking, background serving, automatic unloading and updates. Ollama's FAQ shows how to check whether a model landed on the GPU: ollama ps shows a split like 48 percent CPU and 52 percent GPU if the model spilled into system memory. It makes the common case easy and gives you environment variables for the rest.

The trade is the usual one. Convenience layers decide defaults for you, and defaults can be wrong for your hardware. Control costs time. If you have never hit a limit in Ollama, you do not need llama.cpp directly.

Using them with coding agents

Both give you an OpenAI-compatible endpoint, which is what most agents need.

• llama.cpp: run llama serve and point the agent's base URL at it. The README calls it an OpenAI-compatible API server, and the repository has a llama-server REST API document.
• Ollama: the base URL is http://localhost:11434/v1/. Its docs also describe Anthropic compatibility, which matters for tools that speak that format.

On the agent side, our lists of open-source agents and free agents show which tools accept a custom endpoint. Aider and OpenCode are the usual picks. See Aider for how it connects to a local model.

The limit is the model, not the runtime. A small local model that fits your GPU is much weaker at multi-step coding than a large hosted one, and neither runtime changes that. If you need a bigger model than your hardware can hold, the hosted route in DeepSeek vs Claude or OpenRouter alternatives is a better fit than squeezing harder.

Performance: what we can and cannot say

We did not run benchmarks for this page, and we are not going to quote numbers from forum threads. What the documents let us say is limited to this.

• Both can use GPU acceleration and both support quantised models, so a fair test uses the same model file and the same quantisation level.
• llama.cpp lists more backends, so on unusual hardware, such as an Intel GPU through SYCL or a Qualcomm chip through Hexagon, it may be the only one that works.
• Ollama's number-one practical effect on speed is keeping a model loaded. The keep_alive setting avoids the cold-start load on every request.

To compare them yourself, load the same GGUF in both, use the same prompt and context size, and measure tokens per second for generation and the time to first token separately. Run it three times and ignore the first run, which pays for model loading.

Privacy and data handling

Both run on your machine and keep prompts local by default. Neither needs an account to run a local model.

With Ollama, remember the cloud option. Its FAQ documents OLLAMA_NO_CLOUD to turn cloud features off, so set it if your policy requires fully local use. llama.cpp has no cloud tier at all. It is an engine, so the question does not arise, though the -hf flag does fetch files from Hugging Face on first use, and your network will see that request.

On exposure, both bind to localhost by default as far as we know, and both can be opened to a network.. Put authentication or a reverse proxy in front before you open either one up.

Limits and gotchas

• Fast releases. Both projects ship constantly. Pin the version you tested.
• Naming drift. llama.cpp's commands changed to llama cli and llama serve in the README we read. Older tutorials use llama-cli and llama-server, and many still work. Check which binaries your install provides.
• Memory. A 4-bit model is smaller, not free. Plan RAM or VRAM for the weights plus the context cache.
• Ollama's library lag. A brand-new model may reach llama.cpp by Hugging Face before it appears under a short name in Ollama's library.
• Docker on Mac. Ollama's FAQ says GPU acceleration is not available for Docker Desktop on macOS. Run it natively there.
• Backends are not equal. Support listed in the README does not promise equal speed or stability on every backend.

Who should pick which

A developer trying local models for the first time

Ollama. You will have a working model before you have finished reading the llama.cpp build guide. If it works for you, stop there.

A tinkerer with unusual hardware

llama.cpp. If you have an AMD card, an Intel GPU, a Snapdragon laptop or an old GPU, the backend list is the reason to go direct.

A team shipping a product that embeds a model

llama.cpp as a library or server, since it is an engine you can build into your own software under MIT. Ollama as a local server is also common for desktop products that want the user to manage models.

A team serving many users

Neither is the ideal tool at scale. Read Ollama vs vLLM for the engine built for concurrent load.

Sizing memory before you download anything

Both runtimes load model weights into RAM or video memory, and the file size follows from simple arithmetic: parameters times bits per weight, divided by eight, gives bytes. This is a rough floor, since real files add some overhead and the context cache comes on top.

So an 8B model at 4 bits fits an 8 GB graphics card or a modest laptop with room to spare, and a 70B model at 4 bits needs a workstation or a large unified-memory Mac before you add the cache. The longer your context, the larger the cache, so a model that loads fine can still run out of memory when you paste in a big file. llama.cpp's hybrid mode, which puts some layers on the GPU and the rest on the CPU, is the escape hatch for a model slightly too big for your card, at the cost of speed.

When to move from Ollama to llama.cpp

Most people never need to. Move when one of these happens.

1. A model you want is not in Ollama's library yet. Pull the GGUF straight from Hugging Face with -hf.
2. Your GPU is not on Ollama's list, and llama.cpp lists a backend for it.
3. You need to set layer offload, batch size or context directly, and Ollama's settings do not reach what you need.
4. You are embedding the engine in a product, and want to control the exact build.

Moving back costs little. Both speak an OpenAI-compatible API, so the agent or app only needs a new base URL, from http://localhost:11434/v1/ for Ollama to whichever port llama serve listens on.

If something does not work

• The model loads and answers slowly. Check whether it is running on the GPU or the CPU. A model too large for video memory may be falling back to the CPU.
• Out of memory on a long prompt. Reduce the context size, or pick a smaller quantisation file.
• The agent cannot connect. Confirm the server is running, the port matches the base URL, and the model name in the agent matches the name the server knows.
• Results change after an update. You updated the runtime or the model file. Pin both once something works.
• Ollama still reaches out to the network. Check whether cloud features are on, and set OLLAMA_NO_CLOUD if your policy requires fully local use.

Our take

Treat this as a layers question. llama.cpp is the engine layer and Ollama is the product layer, and the product layer is usually what a person wants. We think the default advice is correct for most readers: use Ollama, and reach for llama.cpp the day you need a flag Ollama does not give you.

We stay honest about what we did not do. We did not benchmark, and we did not verify the relationship between the two codebases from Ollama's own documents. Both are MIT, both are active (each had a push within a day of our check), and both are safe choices. The expensive mistake is not picking the wrong one. It is picking a model too large for your hardware and blaming the runtime.

Checklist

1. Write down your hardware: GPU model, VRAM, system RAM, OS.
2. Install Ollama and run one small model to confirm the machine works.
3. Run the same model through llama.cpp using -hf, and compare memory use.
4. Decide which one your agent or app will call, and set its base URL.
5. Pin versions and keep a note of the quantisation you used.
6. Turn off cloud features and network exposure unless you chose them.

Related pages

• Ollama vs LM Studio
• Ollama vs vLLM
• Open-source LLMs
• Private AI