Ollama: Self-Hosted LLM Runtime, Setup and Limits

Ollama is an MIT-licensed runtime that downloads open-weight language models and runs them on your own computer, then serves them through a local API on port 11434. You install it, run one command, and chat. As of 31 October 2026 the latest release is v0.40.13. It also now sells paid cloud models, which is separate from local use.

At a glance

For other runtimes, read Ollama vs vLLM, llama.cpp vs Ollama and Ollama vs LM Studio.

What Ollama does

Ollama bundles three jobs. It pulls model files from its library. It loads them into memory using a built-in engine. And it exposes them over HTTP. You do not install Python environments or compile anything. That simplicity is its main selling point and the reason it sits behind most "run it locally" tutorials.

The same install also connects coding agents and desktop apps. Its quickstart shows launchers for Claude Code, Codex CLI and OpenCode, and a way to connect Claude Desktop and ChatGPT Desktop on macOS.

Install and first run

Download the app from ollama.com for macOS, Windows or Linux. From a terminal:

ollama run gpt-oss:20b

This pulls the model (14 GB listed for gpt-oss 20B) and opens a chat. Other everyday commands:

ollama pull qwen3-coder:30b   # download without running
ollama list                   # models on disk
ollama ps                     # what is loaded and where it runs
ollama stop gpt-oss:20b       # unload from memory
ollama rm qwen3-coder:30b     # delete

On Linux you use the install script from the docs, and you re-run it to update. On macOS and Windows the app updates itself.

The API

The server starts with the app, or with ollama serve. A native request:

curl http://localhost:11434/api/generate -d '{"model": "gpt-oss:20b", "prompt": "Why is the sky blue?", "stream": false}'

And the OpenAI-compatible path, which lets existing client libraries work:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")

The key is a placeholder. Ollama's compatibility page lists what is supported. The docs also describe an Anthropic-compatible API, which is how Claude Code connects.

Settings that matter

Memory needs scale with OLLAMA_NUM_PARALLEL times the context length, per the FAQ. If three people share one box, this is the line that explains why a model that fits for one person falls over for three. On macOS you set variables with launchctl before opening the app, and on Windows Ollama reads user and system environment variables.

Hardware

Ollama's hardware page is specific:

• Nvidia: compute capability 5.0 and above with driver 550 or newer. Cards at compute capability 5.0 to 6.2 need driver 570. The list runs from GTX 750 up to RTX 50 series, A100, H100, H200 and GB10.
• AMD: Radeon cards through ROCm v7 on Linux, with more cards through Vulkan.
• Apple: Metal acceleration.
• Docker: GPU passthrough works on Linux and on Windows with WSL2. It does not work in Docker Desktop for macOS.

If a model is larger than VRAM, Ollama splits it across GPU and CPU, and ollama ps reports the ratio. For sizing, see GPU requirements for LLMs and our run an LLM locally guide.

Models

The Ollama library lists models with size and context. Two we read on 31 October 2026:

Licences vary by model family, so read each model's page before commercial use.

Using coding agents with Ollama

Ollama has launchers that start an agent already pointed at a model:

ollama launch claude
ollama launch codex
ollama launch opencode

For Claude Code the docs also give a manual route, which sets three environment variables:

export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434
claude --model qwen3.5

For Codex, codex --oss uses a local model and codex --oss -m gpt-oss:120b picks one. Ollama advises a context window of at least 64k tokens for Claude Code, Codex and OpenCode, and at least 32K for Cline. The full walkthrough is in local coding agent with Ollama, and agent profiles are at Claude Code, OpenAI Codex CLI and OpenCode.

Ollama cloud: a different thing

Ollama now sells hosted models. Per its pricing page:

Per-token examples from the same page: gpt-oss:120b at $0.15 per million input and $0.60 per million output, and gpt-oss:20b at $0.07 and $0.30. Off-peak rates apply outside 12:00 to 18:00 UTC on weekdays. This is the place to be careful with the word "self-hosted". Using an Ollama cloud model sends your prompt to Ollama's servers. Using a local model does not.

Privacy

The FAQ answers the question directly. Run locally, and Ollama does not see your prompts or data. Use a cloud model, and Ollama processes prompts and responses to provide the service, does not store or log that content and never trains on it. It collects basic account information and limited usage metadata. If you want no cloud path at all, the FAQ describes a local-only mode that disables cloud features, which also removes cloud models and web search.

Beyond Ollama itself, the exposure is your network. The server has no login. Keep it on localhost, or put a reverse proxy with TLS and authentication in front. Our self-host guide shows the steps.

Running Ollama in Docker

Ollama ships a container image. The CPU-only form from the docs:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

For Nvidia GPUs you first install the NVIDIA Container Toolkit and configure Docker with sudo nvidia-ctk runtime configure --runtime=docker, restart Docker, then add the GPU flag to the same command. Keep the named volume, because it holds the downloaded models and replacing the container without it means downloading tens of gigabytes again. Docker is a good fit for a server because the version is pinned in your compose file and rolled back by changing a tag.

Running it as a shared service

A team setup has four parts, each of which the docs cover.

1. Bind address. Set OLLAMA_HOST=0.0.0.0:11434 only on a host that sits behind a firewall.
2. Proxy. The FAQ shows an Nginx proxy_pass to localhost:11434. Terminate TLS there and add authentication, since Ollama's server has none.
3. Keep-alive. Set OLLAMA_KEEP_ALIVE=-1 so the first request each morning does not wait for a reload.
4. Concurrency. Raise OLLAMA_NUM_PARALLEL only as far as memory allows, and measure with ollama ps while people are working.

If you find yourself tuning the fourth item every week, that is the signal to look at vLLM, which treats batching as its core job.

Ollama against the alternatives

The same licence column explains why many companies pick Ollama or llama.cpp when legal review matters: both are MIT and the source is on GitHub.

An example workload and what it costs

Ollama itself is free locally, so the cost question is about the machine and about the cloud tier.

Local, one developer. The software costs nothing. Memory is the cost. A laptop with 32 GB of memory runs gpt-oss 20B, which is listed at 14 GB, and leaves room for an editor and a browser. If you already own the laptop, the marginal cost is electricity.

Cloud, light agent use. Take an example volume we chose, not a measured one: a coding agent that uses 5 million input tokens and 1 million output tokens in a month on gpt-oss:120b. At the listed rates of $0.15 and $0.60 per million, that is $0.75 plus $0.60, so $1.35 for the month before any plan fee. The Pro plan at $20 includes $60 of credits, so a user at this volume would not use them up. Heavy agent loops use far more tokens, so measure your own before you pick a plan.

Cloud, a team. Team is $500 a month with $1,000 of shared credits. That only makes sense if your team would really spend more than $500 in tokens, or needs central billing and unlimited seats. At lower spend, pay as you go is the cheaper plan.

For the same model on other hosts, see our hosting hub and the interactive hosting compare. Cloudflare Workers AI lists gpt-oss-120b at $0.35 and $0.75 per million tokens, and Groq lists it at $0.15 and $0.60.

Upgrading, pinning and rollback

Ollama ships often. The checked release, v0.40.13, was published on 28 October 2026. On macOS and Windows the app downloads updates itself, which is convenient for one person and a risk for a team box. For a shared server, install from the script or container tag you tested, and upgrade deliberately on a quiet day. Keep the previous container tag so that rollback is a one-line change. After any upgrade, run ollama ps and a short prompt against each model your team uses, because model files and defaults can change between versions.

A setup checklist

• Latest stable release installed, or a pinned tag in Docker.
• Models pulled from the library and licences read.
• Context length set for your main use case, 64k for coding agents.
• ollama ps shows the model fully on the GPU.
• Port bound to localhost, or behind a proxy with authentication.
• Cloud features disabled if you want local-only.
• A named volume or backup for the model directory.
• A written fallback if the box is down.

Troubleshooting

The model answers slowly. Run ollama ps. If the processor column shows a CPU share, the model does not fit in VRAM. Pick a smaller or more compressed model, or reduce the context length.

The agent loses track of the project. The context window is probably too small. Ollama's docs recommend 64k or more for Claude Code, Codex and OpenCode. Raise OLLAMA_CONTEXT_LENGTH and restart.

The GPU is ignored on Linux after the laptop slept. The docs describe a driver bug after suspend and resume, with the workaround sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm.

Docker on a Mac cannot use the GPU. The docs say there is no GPU passthrough on Docker Desktop for macOS. Run Ollama natively on the Mac.

A teammate cannot connect. Check the bind address and the firewall. Ollama listens on 127.0.0.1 unless you change OLLAMA_HOST, and browser clients also need OLLAMA_ORIGINS for their origin.

Where Ollama falls short

• Shared servers. It handles one-to-few users well. For many concurrent users, vLLM is built for throughput.
• Fine control. If you need to tune the engine itself, llama.cpp exposes more. Ollama's Modelfile reference lists context size, sampling and prediction parameters, but no per-layer GPU offload or batch size setting.
• Small default context. The VRAM-based defaults are low on small machines. Raise them on purpose.
• Apple speed. We have not compared Ollama with LM Studio on Apple silicon, so time both on your own Mac before you commit.

Who should use it

Our take

Ollama earns its place by being the shortest path from nothing to a working local model. The MIT licence, the OpenAI-compatible API and the agent launchers cover most first projects. The cloud tiers are a real product but a different one, and a page that says "self-hosted" should not blur them. Use local models for privacy and learning, cloud models when you need bigger ones, and move to vLLM when the box gets busy. Our conclusions come from the docs and pricing page, not a hands-on test.

Related pages

• Private AI hub and self-hosted AI overview
• How to self-host an LLM
• Hosting hub for rented GPUs