Self Hosted AI: What It Takes to Run Your Own

Self-hosted AI means you run an open-weight model on hardware you control and send prompts to your own endpoint, not to a vendor. The fastest start is Ollama on a laptop. A team usually moves to a GPU server running vLLM. The real decision is whether the privacy or the volume justifies the work, and we checked the numbers on 31 October 2026.

The short version

If you want the step-by-step install, our guide on how to self-host an LLM has the commands. This page is the decision layer that sits above it: what the pieces are, what they cost and where people get stuck.

What "self-hosted" covers

People use the word for three different setups, and the costs differ a lot.

1. A machine on your desk. A laptop or workstation runs the model for you. Nobody else uses it. Cost is hardware you already own.
2. A server your team shares. One box with one or more GPUs sits in an office or a rack. Several people send requests at once. Cost is the hardware plus someone's time.
3. A rented GPU you manage. You rent a GPU by the hour from a cloud, install the runtime yourself and keep the weights. Cost is an hourly rate. This is self-hosting in the sense that you choose and control the software, but the hardware belongs to someone else. See Runpod and Lambda Cloud.

A fourth thing is often confused with these: a hosted API that serves open models, such as Groq or Cerebras. You do not run anything there. We cover the difference in cloud vs self-hosted AI.

The pieces of a self-hosted stack

Model weights

Open-weight means the trained parameters are published for download. They come with a licence you should read. For example, OpenAI's gpt-oss models are listed on Ollama with an Apache 2.0 licence, which the model page describes as permissive, with no copyleft restrictions. Qwen3-Coder from Alibaba is also listed for local use. Always read the licence file on the model's page before a commercial rollout, since terms differ between families. Our open vs closed models page covers that choice.

A runtime

The runtime loads the weights and answers requests. Three are common.

The vLLM docs say it does not support Windows natively and point Windows users to WSL. Ollama's hardware page lists Nvidia GPUs from compute capability 5.0 up, AMD cards through ROCm and Vulkan, and Apple GPUs through Metal. The llama.cpp README lists Apple silicon as a first-class target, with CUDA, HIP, Vulkan and SYCL backends.

For a longer side-by-side, read Ollama vs vLLM and llama.cpp vs Ollama.

An API your tools can call

All three expose an OpenAI-style HTTP API. Ollama listens on port 11434 and vLLM on port 8000 by default. That means most chat front ends, agent frameworks and coding tools can point at your server by changing a base URL and a dummy key. Ollama also documents an Anthropic-compatible endpoint, which is how it connects to Claude Code.

A network setup

By default Ollama binds to 127.0.0.1, so only your own machine can reach it. The moment you widen that, you are running a service that needs a proxy, TLS and authentication. Ollama's local server has no login step. vLLM can check a key if you pass --api-key. Treat the port like a database port.

How much hardware you need

Weights take roughly parameters times bytes per parameter. A 4-bit build uses about half a byte per parameter. Add room for the context cache, which grows with the conversation and with the number of parallel requests.

Published sizes from the Ollama library show the real numbers:

The gpt-oss figures work because OpenAI post-trained the models with the mixture-of-experts weights in MXFP4, about 4.25 bits per parameter, per the model page. That is why a 120B model fits an 80 GB card. Our GPU requirements for LLMs page goes through VRAM by size and quantisation in more detail.

A practical reading: a modern laptop with 16 to 32 GB of memory runs a 20B-class model for one person. A single 80 GB GPU handles a 120B-class model. Anything near 480B is a multi-GPU or large unified-memory machine, and at that point renting is usually more sensible than buying.

What it costs

You pay in one of three ways.

Hardware you own

A workstation GPU is a one-off purchase plus power and your time. The break-even depends on what you would otherwise spend on API calls. Our worked example in how to self-host an LLM shows a case where $400 a month of API spend and a $3,000 machine needs about 22 months to pay back. Those are example figures, not quotes. The pattern is the useful part: at modest API spend, owning rarely wins on cost alone.

A GPU you rent

Prices checked on 31 October 2026:

Some links on this page go through /go/ and may earn us a fee; it never changes our order. For a worked monthly comparison of these providers, see Runpod vs Lambda, Modal vs Runpod, and the interactive hosting compare.

An example, with volumes we chose for illustration: an H100 at $3.99 an hour, run for 8 hours a day over 22 working days, is 176 hours, or about $702 a month. Left on all month it is about $2,913. The gap is why renting works for part-time or bursty use and gets expensive when idle.

A hosted API instead

Hosted open-model APIs charge per token and have no idle cost. Groq lists gpt-oss-120b at $0.15 per million input tokens and $0.60 per million output tokens. Cloudflare Workers AI lists the same model at $0.35 and $0.75. At low and medium volume this beats any server you run, which is why cloud vs self-hosted is a volume question first.

Privacy and data control

What you gain by self-hosting is simple to state. Prompts and outputs never leave your network unless you send them. There is no vendor retention policy to read, no training-use clause, and no third-party outage to wait out.

Ollama's FAQ says it runs locally and does not see your prompts or data when you run locally. It also sells cloud models now, and for those it says it processes prompts and responses to provide the service but does not store or log that content and never trains on it. The same FAQ says you can run Ollama in local-only mode by disabling its cloud features. If privacy is the reason you are here, turn that off and confirm it.

What you take on in return:

• Access control for the port, and a log policy you write yourself.
• Patching the runtime, the driver and the host.
• Provenance of the weights. Download from the publisher's own page or repository, and check the hash if one is given.
• Backups of anything you fine-tune.

Self-hosting removes one risk, the vendor, and adds another, your own operations. Whether that trade is good depends on who is operating it.

Coding agents on a self-hosted model

Several coding agents can point at a local endpoint. Ollama's own docs describe ollama launch claude, ollama launch codex and ollama launch opencode, and tell you to give Claude Code, Codex and OpenCode a context window of at least 64k tokens. Cline's page asks for at least 32K. We cover each agent's configuration in local coding agent with Ollama and self-hosted AI coding assistant. The agents themselves are listed under AI coding agents.

Be honest about quality. Open models of this size are improving, but long tasks that touch many files are where a local 20B model most often loses to a frontier API. Try your own tasks before you commit a team.

Limits and gotchas

• Default context is small. Ollama picks a default from VRAM: 4k under 24 GiB, 32k from 24 to 48 GiB, 256k at 48 GiB or more, per its docs. A coding agent that reads files needs more than 4k.
• Memory spills silently. When a model does not fit the GPU, Ollama splits it across GPU and CPU. ollama ps shows the split, and speed drops.
• One model at a time on vLLM. A vLLM server hosts one model per process by default.
• Concurrency is a setting. Ollama's OLLAMA_NUM_PARALLEL defaults to 1, and memory needs scale with it.
• Upgrades change behaviour. Runtimes and model files move fast. Pin versions for a team.

Who should self-host

Mistakes we see in plans for self-hosted AI

Most failed rollouts trace back to the same few choices, and all of them can be avoided on paper before any money is spent.

Buying hardware before measuring demand. Teams often size a server for the model they wish to run, not the one they need. Count how many people will send requests at the same moment, because concurrency, not headcount, sets the GPU memory you need. One person chatting uses the model in bursts. A coding agent that loops through tool calls keeps it busy for minutes at a time.

Ignoring the context cache. Weights are the number everyone quotes. The key-value cache is the number that breaks the plan. Ollama's FAQ ties required memory to context length and parallel requests together. Double the context and the cache roughly doubles, so a model that fits at 8k can fall off the GPU at 64k.

Treating the port as private because it is inside the office. A flat office network, a VPN with many users and a shared Wi-Fi are all places where an open inference port gets used by people you did not plan for. Put authentication in front of it from day one.

Skipping a fallback. Hardware fails and drivers break after updates. Decide in advance whether the team falls back to a hosted API, to a rented GPU, or simply waits. If the data is the reason you self-host, the fallback has to respect the same rule, so write that down.

Expecting one model to do everything. A small fast model is right for autocomplete and quick questions. A bigger one is right for planning and review. Many teams run two models and route by task. That is a cost question as much as a quality one, and it is easy to test with a few days of real prompts.

Our take

From the documented facts, self-hosting is a good fit when one of two things is true. Either your data may not leave, or your volume is high and steady enough that a fixed machine beats per-token billing. If neither holds, the engineering time usually costs more than the API. If one holds, start small: a single machine, one runtime, one model, and a clear owner. Add a second server only when the first is busy.

We have not run a head-to-head test of local and hosted models for this page, so we make no claim about relative output quality beyond what the vendors state.

A first-week plan

1. Pick one use case, for example a private chat for the team or a coding agent.
2. Install Ollama on a machine that meets the memory figure for a 20B-class model, and pull one model.
3. Set the context length for the use case, then check with ollama ps that the model sits fully on the GPU.
4. Decide who may reach the port. Keep localhost unless you need more, and put a proxy in front if you do.
5. Run real prompts from your own work for a week and note where it fails.
6. Only then price a server. Use the figures above, and your own usage, in the hosting compare.

Related pages

• Private AI hub and hosting hub
• Run an LLM locally
• Ollama and vLLM
• Private AI for teams
• On-premise AI