On-Premise AI: Run Models Inside Your Own Walls

On-premise AI means the models, the GPUs and the data all sit on hardware you own, in a room or rack you control, so no prompt crosses your network edge. You run an open-weight model with a server such as vLLM or Ollama. It costs more upfront than renting and pays off only at steady, high use or when rules forbid leaving. Facts here were checked on 31 October 2026.

When on-premise is the right word

The label is stretched in marketing. Use it for one thing: you own or lease the physical servers and they sit in space you control. Here is how the neighbouring setups differ.

Our cloud vs self-hosted AI page puts these on one spectrum. This page is about the first two rows, and about when the next rows would serve you as well.

Why organisations choose it

Data cannot leave. Some data sits under contracts, laws or internal policy that bar sending it to a third party.. On-premise removes the question.

Steady, heavy use. A GPU you own has no hourly meter. If it runs most hours, the purchase price spreads thin.

Latency and locality. Models next to your data and your users avoid a round trip to a distant region.

Independence. No vendor can change a price, a model or a policy under you.

Air-gapped sites. Some environments have no internet access at all. Only on-premise works there. Note that the install steps in the docs assume downloads from the network, so you would need an offline process for weights and packages.

The software stack

The software is free and open source.

Pick the server by users. For under five people Ollama is fine. For a department, use vLLM. The Ollama and Ollama vs vLLM pages go deeper.

Hardware you will meet

Our GPU requirements for LLMs page tabulates sizes. For planning on-premise, the points that matter are these.

Memory first. The model and its cache have to fit. One 80 GB card holds gpt-oss 120B (65 GB listed). A 480B coding model needs 250 GB or more, per its page.

Card families. Ollama's hardware page lists support for Nvidia cards from compute capability 5.0 up through RTX 50, A100, H100, H200 and the GB10 in the DGX Spark, plus AMD cards through ROCm and Vulkan. vLLM supports NVIDIA CUDA, AMD ROCm 6.3 or above, Intel XPU, TPU and Intel Xeon CPU, with Apple silicon through a community plugin.

Power and cooling. Data-centre GPUs draw hundreds of watts each, and a multi-GPU server needs a circuit and cooling to match.

Networking. A single server needs only a good switch port. Multi-node serving, which vLLM supports with tensor and pipeline parallelism, needs fast links between nodes.

Spares. One server is a single point of failure. Decide whether downtime is acceptable or whether you need a second node.

What it costs

Own versus rent is a calculation with four inputs: purchase price, years of life, how many hours a month the GPU actually runs, and the people cost of operating it.

Prices to compare against, checked on 31 October 2026. Some links on our site go through /go/ and may earn us a fee; it never changes our order.

Notice that the reserved table at Lambda is higher per hour than its on-demand line. The reserved tables on that page cover clusters from 16 GPUs, which is a different product from a single instance.

An example with figures we chose, not quotes. Suppose a team would otherwise rent an H100 at $3.99 an hour for 300 hours a month. That is $1,197 a month, $14,364 a year. If a purchased server with a comparable GPU costs a six-figure amount (example), the payback is many years, unless usage rises or the team needs several machines. If the same team ran the GPU 600 hours a month, the rental bill doubles and ownership looks better. Compare using your own quote.

People cost is the item most plans leave out. Someone must patch the host, update the runtime, watch the logs and answer when it breaks. A few hours a month is a reasonable floor for one server. Price those hours.

A simple rule from our self-host guide: if the break-even is beyond two years, the hardware will be near the end of its useful life when it pays back. Think hard before buying then.

Security duties you take on

Keeping data on-premise does not make it safe by itself. What shifts to you:

1. Network exposure. Neither Ollama nor vLLM, beyond its optional API key, has a user login. Put the server on an internal network segment with a proxy in front.
2. Authentication and authorisation. One key per user or app, rotation, and removal when people leave.
3. Logging and retention. Decide what is logged. Prompt logs store your sensitive data a second time.
4. Patching. The OS, the GPU driver, the runtime and the container images all need updates.
5. Model provenance. Download weights from the publisher, record the version and hash, and scan any custom code the model loads.
6. Physical security. The server is data. Lock the room.
7. Backups. Anything you fine-tune or configure needs a copy.
8. Incident plan. Know whom you call and what you cut off.

Ollama's FAQ gives one example of the boundary question: with cloud features enabled, Ollama processes prompts for its cloud models, and it offers a local-only mode that disables them. On an air-gapped or regulated server, disable cloud features and confirm no outbound path with your firewall.

Compliance: what the vendors say and what they do not

The docs we read describe technical controls, not legal status. For example, AWS's Bedrock documentation says model providers do not have access to the deployment accounts or to customer prompts and completions. That is a statement about Bedrock's architecture. It is not a compliance certificate for your use. For any managed alternative, ask for the data processing terms and the audit reports you need, and have counsel read them.

On-premise lets you meet strict data-location rules by construction, since the data never moves. It does not give you certificates for free. You still need your own controls, evidence and audits.

Sizing by organisation

These tiers are our planning shapes, built from the listed model sizes and documented runtime behaviour. They are not vendor figures.

The first row runs on what the self-host guide covers. The last row is a different project, with procurement, security review and a team. Do not borrow the last row's plan for the first row's problem.

Setting up without internet access

An air-gapped install changes how you get software, not what you run. The vLLM and Ollama quickstarts assume network downloads, for packages, container images and model weights, so plan an offline path.

1. On a connected machine, download the container image or Python wheels for a pinned version of the runtime, and the model files from the publisher's page. Record the versions and checksums.
2. Move them across on approved media, through your usual transfer control.
3. On the isolated host, load the image or install the wheels, place the model files where the runtime expects them, and start the server with no outbound route.
4. Disable anything that phones home. For Ollama that means turning off cloud features, which the FAQ describes. Confirm with a firewall rule that denies outbound traffic and a test that the server still works.
5. Plan for updates. Each upgrade repeats the transfer. Batch them on a schedule.

vLLM downloads precompiled kernels with vllm download-kernels, and loads weights from Hugging Face by default, so both need an offline equivalent.

Utilisation: the number that decides everything

A GPU you own earns its keep by running. Take a worked example with numbers we chose. A server runs for 8 working hours a day on 22 days. That is 176 hours a month, a utilisation of about 24 percent of the 720 hours in a month. Renting the same capacity for those 176 hours at $3.99 an hour costs about $702, and owning pays for 720 hours of power and space whether used or not. Push use to three shifts or batch jobs overnight and the same machine is busy for 500 or more hours, and rental for those hours is about $1,995.

So the questions to ask before buying are:

• How many hours a month will the GPU be busy, honestly?
• Can overnight batch work fill idle hours, such as indexing, summarising or evaluation runs?
• Will demand grow, or has it levelled off?

If the answers are "few hours, no batch work, flat", rent. If they are "most hours, plenty of batch work, growing", own.

Roles you need

A small on-premise deployment still needs named owners.

In a small company one person may hold all five, which is workable but fragile. Write the roles down anyway, so a holiday does not become an outage.

Procurement checklist

• Model and context target defined, with the listed size, plus 25 percent for cache.
• GPU memory sized to that figure, with a plan for what happens at double the users.
• Driver and runtime support confirmed in the Ollama or vLLM docs for the exact card.
• Power, cooling and rack space confirmed by whoever runs the room.
• Warranty and spare-part terms read.
• A software plan: runtime, proxy, monitoring, backups.
• A security review booked before go-live.
• A fallback for downtime, with its data terms checked.
• A rent-first test completed, so the purchase rests on measurements.

When a managed alternative is enough

Many teams that say "on-premise" need only one of its properties. Check which.

Use the interactive hosting compare and the hosting hub to filter by region and data terms.

A phased plan

1. Prove the use case on rented GPUs. A week on a Runpod A100 at $1.79 an hour costs about $300 if left on around the clock, and gives you real concurrency and memory numbers.
2. Write the data policy. Which data may be sent where.
3. Build a pilot on one server. vLLM, a proxy, per-user keys, monitoring.
4. Run it with real users for a quarter. Measure utilisation and failure.
5. Decide to scale, to buy more, or to move to a managed host. The numbers from step 4 settle it.

Our take

On-premise AI is a good answer to a narrow question: data that cannot leave, or use steady enough to beat rental. For everyone else it is an expensive way to feel safe. If you are unsure, rent first, measure for a quarter and buy only what the measurements justify. Choose open, permissively licensed software so you are never locked in, and treat the server as production infrastructure from day one. These views come from vendor documentation and pricing pages, and we did not test hardware.

Related pages

• Private AI hub and Self-hosted AI
• Private AI for teams
• How to self-host an LLM
• Hosting hub