How to Self-Host an LLM

By the end of this guide you will have an open-weight language model running on hardware you control, answering requests through an API your own apps can call. The shortest path is Ollama: install it, run ollama pull, and send a request to localhost:11434. That takes about fifteen minutes on a laptop. Step 6 covers moving to vLLM when several people need it at once. Facts checked 31 October 2026.

What you will have at the end

• A local model server with a clear start, stop and update routine
• An OpenAI-style endpoint your scripts and agents can point at
• A rule for which model size fits your memory
• A network setup that does not leave the server open to the internet
• A cost picture for laptop, owned server and rented GPU

If you are still deciding whether to host at all, read cloud vs self-hosted AI first. For the wider picture of private setups, see self-hosted AI.

Prerequisites

Pick the hardware path before you install anything.

Ollama's quickstart gives one concrete sizing point. Its local example, Gemma 4 E2B, is about a 7.2 GB download, and the docs recommend 8 GB of available VRAM, or unified memory on a Mac. Larger context windows need more memory. With less VRAM, Ollama can use system RAM, but responses may be slower.

For NVIDIA cards, the hardware page lists support for compute capability 5.0 and above with driver 550 or newer. Cards from 5.0 through 6.2 need driver 570 or newer. Check your card on NVIDIA's compute capability list before you plan around it.

You also need:

• Free disk space for models. Several gigabytes each is normal.
• A terminal, and curl for the test calls.
• Docker, only if you want the container route.
• Python 3.11 or newer on Linux, only for the vLLM step.

Step 1: Install Ollama

On macOS and Windows, download the app from ollama.com/download and open it. On Linux, one line installs it:

curl -fsSL https://ollama.com/install.sh | sh

If you are upgrading an older Linux install, the docs tell you to remove the old libraries first with sudo rm -rf /usr/lib/ollama. On Linux, start the server with ollama serve if it is not already running.

Ollama on macOS and Windows downloads updates for you. You click "Restart to update" to apply them.

Step 2: Pull a model and talk to it

Download a small model first, so you learn the loop before you spend an hour on a big file.

ollama pull gemma4:e2b
ollama run gemma4:e2b

ollama run opens a chat in your terminal. Type a question, then exit. That is a working self-hosted LLM. Everything after this is about making it useful and safe.

Which model should you actually run? Our open-source LLM list and the best open-source LLM for coding page compare families such as Llama, Qwen and Gemma. A simple rule: pick the largest model whose download fits comfortably in your VRAM, with room left for the context window.

Step 3: Call it from code

Ollama listens on 127.0.0.1:11434 by default. This is its native chat endpoint:

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:e2b",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "stream": false
  }'

The reply text is in message.content. Ollama also supports a subset of the OpenAI API, so most OpenAI client libraries work if you change the base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
    model="gemma4:e2b",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(reply.choices[0].message.content)

The key can be any string, because a local server does not check it. The compatibility page warns that OpenAI support is a subset of the full API, so test the features you rely on, such as tool calling and structured output, before you switch an app over.

Step 4: Fix the settings that bite

Three defaults decide whether your first week is pleasant.

Context length. The docs say Ollama picks a default context length from your VRAM: 4k tokens under 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. Four thousand tokens is small for a coding agent that reads files. A larger window needs more memory, so raise it only as far as your hardware allows.

Keep-alive. A model unloads after a period of inactivity. Set how long it stays in memory with the keep_alive request parameter, or server-wide with OLLAMA_KEEP_ALIVE. Using -1 keeps it loaded.

curl http://localhost:11434/api/generate -d '{"model": "gemma4:e2b", "keep_alive": -1}'

Parallel requests. OLLAMA_NUM_PARALLEL sets how many requests each model handles at once, and it defaults to 1. The FAQ notes that required RAM scales by OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH. If three people share a box, this setting is where you feel it. OLLAMA_MAX_LOADED_MODELS caps how many models sit in memory at once.

Step 5: Use it with a coding agent

You can point a coding agent at your own server. Ollama's quickstart shows a launcher for this:

ollama launch claude
ollama launch codex
ollama launch opencode

Each command starts the named agent against an Ollama model, and offers to install the agent first for some of them. Pick a model when prompted. See Claude Code, OpenAI Codex CLI and OpenCode for what each agent is, and local coding agent with Ollama for the longer setup.

Be realistic about quality. A small local model is slower and less capable than a hosted frontier model on long, multi-file tasks.

Step 6: Move to vLLM for a shared server

Ollama is built for ease. When many people send requests at once, vLLM is the usual next step. Its docs list Linux and Python 3.11 through 3.14 as prerequisites, with Apple Silicon support through a separate vLLM-Metal path. For NVIDIA GPUs the quickstart uses uv:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

The download-kernels step installs precompiled kernels that vLLM would otherwise download and compile at startup, which the docs say can add several minutes on Hopper and newer GPUs. Then start a server:

vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key YOUR_LONG_RANDOM_KEY

By default it listens on http://localhost:8000, hosts one model at a time and speaks the OpenAI protocol. Check it with the models endpoint:

curl http://localhost:8000/v1/models -H "Authorization: Bearer YOUR_LONG_RANDOM_KEY"

Two things from the docs matter. First, --api-key (or the VLLM_API_KEY variable) makes the server check a key, and you can pass several keys for rotation. Second, by default vLLM applies the generation_config.json from the model's Hugging Face repo, so sampling defaults can differ from vLLM's own. Pass --generation-config vllm if you want vLLM's defaults instead.

For a side-by-side of the two servers, see Ollama vs vLLM and our vLLM page.

Step 7: Put it behind a safe network setup

This is the step people skip. By default Ollama binds 127.0.0.1, so only your machine can reach it. That is safe. The FAQ shows how to widen it by setting OLLAMA_HOST, for example 0.0.0.0:11434, and also says you can put it behind a proxy such as Nginx.

The local server docs describe no login step. Assume anyone who can reach the port can use your GPU and your models. So:

1. Keep the default bind unless another machine really needs access.
2. If it does, put a reverse proxy in front, with TLS and authentication, and firewall the raw port.
3. On vLLM, always pass --api-key when the port is reachable from anywhere but localhost.
4. Allow cross-origin requests only from origins you trust. Ollama allows 127.0.0.1 and 0.0.0.0 by default, and OLLAMA_ORIGINS adds more.
5. Do not publish the port on a public IP "just for a minute".

Step 8: Run it in Docker if you prefer containers

Ollama's own container works with or without a GPU.

# CPU only
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

For NVIDIA GPUs you first install the NVIDIA Container Toolkit, then configure Docker to use it:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The docs then show the same docker run with the GPU flag added. The named volume (-v ollama:/root/.ollama) keeps your downloaded models when the container is replaced, so do not drop it.

Sizing a model by arithmetic

You can estimate whether a model fits before you download it. A model's weights take roughly the parameter count multiplied by the bytes stored per parameter. The number of bytes depends on how the weights were compressed, which is called quantization. These are rules of thumb, not figures from Ollama or vLLM, and real files add overhead.

Then add room for the context window. The model keeps a key-value cache for every token in the conversation, and it grows as the conversation grows. That is why the Ollama FAQ ties required memory to context length and to the number of parallel requests. A practical rule is to leave 20 to 30 percent of your memory free beyond the weights, and more if you raise the context length. If a model "just fits" it will crawl the moment a long file lands in the prompt.

Lower precision saves memory and costs some quality. Four-bit builds are the usual choice on a laptop. On a rented GPU with room to spare, a larger model at 8-bit often beats a smaller one at 16-bit. Test on your own prompts, because published benchmarks measure other tasks. Model cards on the publisher's page list the sizes and licences on offer, and our GPU requirements for LLMs page goes further.

Commands you will use every week

If ollama ps shows a model split between GPU and CPU, it did not fit in VRAM, and that explains slow replies better than any setting. Models are large, so set a habit of removing the ones you no longer use. Disk fills quietly.

Updating without surprises

Treat a model server like any production service, even a small one.

1. Pin what you depend on. A tag such as gemma4:e2b can point at a new build later. Note the tag and the date you tested it, and if an app depends on exact behaviour, record the digest that ollama show prints.
2. Update on a schedule, not on a whim. Pull a new model on a test machine first, run your test questions, then move it to the shared server.
3. Keep the old one for a week. Disk is cheaper than a broken morning.
4. Read release notes for the server itself. The Linux docs note that older installs need /usr/lib/ollama removed before upgrading, which shows that upgrades can need a manual step.
5. Restart in a quiet hour. An update interrupts running requests.

Sharing one server among a team

A shared box needs a few rules that a laptop does not.

• One entry point. Give people a single URL behind a reverse proxy with authentication, and do not hand out the raw port.
• Per-person keys. On vLLM, --api-key accepts several keys, so you can give each user one and revoke it without a restart cycle for everyone. Ollama has no built-in login, so the proxy has to do this job.
• A queue, not a surprise. With OLLAMA_NUM_PARALLEL at 1, the third person waits for the first two. Decide whether that is acceptable, or raise the setting and pay for the memory.
• Logs with a purpose. Log who called and when. Do not log prompts unless you must, because the logs then hold exactly the data you wanted to keep private.
• A fair-use note. One person running a long agent task can slow everyone. Say so in writing before it becomes a quarrel.

If the team has more than a handful of regular users, that is the point where vLLM, which the docs describe as built for serving many requests, earns its extra setup.

A worked break-even, with example numbers

Self-hosting is a spending decision, so do the sum with your own figures. Take an illustration, not a quote: a team spends $400 a month on a hosted API. A used GPU workstation costs $3,000 to buy, uses about $25 a month in power, and takes four hours a month of someone's time to maintain. Price that time at $60 an hour and the monthly running cost is about $265.

Three cautions apply. A local model of that size will not match a frontier hosted model on hard tasks, so the saving may come with a quality cost that the table hides. Hardware ages, so 22 months is most of its useful life. And if the real driver is privacy, not money, the table is the wrong test: the question is whether the data may leave at all. If your numbers look like these, hosted APIs are probably cheaper. If the hosted spend is several times higher, or the data cannot leave, the sum flips. The cloud vs self-hosted AI page walks through this comparison in more detail.

What can go wrong

What it costs

Software first: Ollama and vLLM are the tools you download. Ollama is MIT licensed and vLLM is Apache 2.0, according to their GitHub repositories. Models carry their own licences, and some limit commercial use, so read the model card before you ship.

The real costs are hardware and time:

• Your own laptop or desktop: no extra bill, but the machine is busy while a model answers.
• A server you own: the GPU, power, and the hours you spend on drivers, updates and monitoring.
• A rented GPU: you pay per hour while it runs. Rates change often. See RunPod, Modal, Lambda and GPU requirements for LLMs.
• Ollama cloud models: the quickstart also offers cloud models with an API key. That is a hosted route, so your data leaves your machine. Leave it out if privacy is your reason for self-hosting.

A fair way to compare: add up a month of hosted API spend for the same work, then price the hardware and your time over a year. See LLM API prices for the hosted side. Self-hosting tends to pay off when use is steady and high, or when the data cannot leave. It tends not to when use is occasional.

Privacy note

Self-hosting keeps prompts and outputs off a vendor's servers, which is the main reason to do it. It does not make you private by itself.

• Check what logs your proxy and server keep, and for how long.
• Download weights from the publisher's official repository, not a mirror you cannot verify.
• Keep the box patched. A server on your network is a target like any other.
• If you also use Ollama cloud models, those requests go to Ollama, so keep them separate from sensitive work.

For regulated data, read private AI for teams and talk to your compliance owner before you rely on a blog guide.

Checklist

• Chose laptop, owned server or rented GPU on purpose
• Model download fits in VRAM with room for context
• ollama run answers a question
• An API call works from code
• Context length and keep-alive set for your use
• Server bound to localhost, or proxied with TLS and auth
• vLLM started with --api-key if you use it
• Model licence read for your use
• A monthly cost figure written down, compared to hosted API spend
• An update routine and an owner named

What this guide uses