vLLM: Serving Open Models to a Team, Setup and Limits

vLLM is an Apache 2.0 inference server that serves open-weight language models from GPUs through an OpenAI-compatible API. It is built for throughput: many users at once on the same hardware. Install it on Linux, run vllm serve <model>, and your apps call port 8000. The latest release we saw is v0.32.0, from 18 October 2026.

At a glance

Compare it with the easy option in Ollama vs vLLM. For the one-machine route, see Ollama.

What problem it solves

A language model generates one token at a time, and each request holds a chunk of GPU memory for its cache. A naive server handles requests one after another or wastes memory on cache it reserves but does not use. vLLM's docs describe two core techniques.

PagedAttention. Memory for the key-value cache is managed in pages, in the way an operating system manages RAM. That lets the server pack more concurrent requests into the same VRAM.

Continuous batching. New requests join the running batch as others finish, so the GPU is rarely idle. The docs also list chunked prefill and automatic prefix caching, which reuses work when many requests share the same prompt start, such as a long system prompt.

The practical effect is more requests per second from the same card, which is why vLLM turns up wherever a team shares a model. The docs link a vendor blog that claims large throughput gains from continuous batching. We do not repeat the number, since it depends on workload.

Install

The docs list Linux, and Python 3.11 through 3.14. For NVIDIA GPUs the quickstart uses uv:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

The download-kernels step installs precompiled kernels that would otherwise be fetched and compiled at startup, which the docs say can add several minutes on Hopper and newer GPUs. AMD GPUs need ROCm 6.3 or above, with prebuilt wheels available. Intel GPUs have basic inference and serving support. On Apple silicon, the docs point to vLLM-Metal, a community-maintained plugin that uses MLX as its compute backend. On Windows, use WSL.

A container route also exists, and most teams prefer it for a server because the version is pinned in a tag. See the Docker section of the install docs for the current image name and flags.

Serve a model

vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key YOUR_LONG_RANDOM_KEY

This starts an OpenAI-compatible server on http://localhost:8000, hosting one model. Check it:

curl http://localhost:8000/v1/models -H "Authorization: Bearer YOUR_LONG_RANDOM_KEY"

Call it from Python with the standard OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR_LONG_RANDOM_KEY")

Two behaviours from the docs matter. First, --api-key, or the VLLM_API_KEY variable, makes the server check keys, and you can pass several. Second, vLLM applies the generation_config.json from the model's Hugging Face repository by default, so sampling defaults may differ from what you expect. Pass --generation-config vllm for vLLM's own defaults.

Tool calling for agents

Coding agents and many apps need the model to call tools. vLLM supports named function calling, plus auto, required and none for tool_choice (the last two as of version 0.8.3). To enable automatic tool choice you pass flags and a parser that matches the model:

vllm serve <model-id> --enable-auto-tool-choice --tool-call-parser <parser>

The docs show llama3_json in a Llama example and list parsers for Hermes-format models (hermes) and OpenAI's open models (openai). A mismatch is the usual cause of tool calls that print JSON instead of acting, so confirm the parser on the tool calling page for your model. See self-hosted AI coding assistant.

Multi-GPU

When a model does not fit on one card, the parallelism docs give the recipe:

The docs advise increasing GPUs and nodes until memory is enough. Multi-GPU buys capacity more than speed, so use it when the model needs it. For sizing, see GPU requirements for LLMs.

Other features

The docs list quantised model support across FP8, MXFP8 and MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ and GGUF, along with compressed-tensors and others. They also list speculative decoding, multi-LoRA serving for dense and mixture-of-experts models, and structured outputs. Which of these your model supports is a per-model question, so check the model card and the vLLM supported-models list.

What it costs to run

vLLM is free. The GPU is the bill. Prices checked on 31 October 2026:

The 220 hours is an example we chose: 10 hours a day on 22 working days. Left on 24/7 for 30 days, 720 hours, a Runpod Secure A100 is about $1,289 and a Lambda H100 is about $3,089. Some links go through /go/ and may earn us a fee; it never changes our order. For a hosted open-model API instead of a server, Groq lists gpt-oss-120b at $0.15 per million input tokens and $0.60 per million output tokens. See Runpod, Lambda Cloud, Modal and the hosting compare.

vLLM next to Ollama and llama.cpp

More: Ollama vs vLLM and llama.cpp vs Ollama.

Planning capacity before you rent

Capacity planning for a vLLM box has three inputs: the model's weights, the cache per request, and the number of requests you expect at the same moment.

Weights. Use the listed download size as a floor. gpt-oss 120B is listed at 65 GB on Ollama and fits a single 80 GB GPU per its model page, with its mixture-of-experts weights stored in MXFP4. gpt-oss 20B is listed at 14 GB. Qwen3-Coder 30B is 19 GB.

Cache. Whatever memory is left after the weights is the cache budget. On an 80 GB card running the 65 GB model that is roughly 15 GB before overhead, and you also reserve some for activations. vLLM's paged cache uses what is available more efficiently than a naive server, but it cannot create memory.

Concurrency. Count simultaneous requests at peak, not users. Ten developers in chat mode might produce two or three live requests at once. Ten developers running agents might produce eight or nine, each with a long context.

A worked example with numbers we chose, not measurements: a team of eight, half using an agent. At peak, four long agent requests run together. On one 80 GB card with the 120B model, the cache is tight, so you either cap context, reduce parallel requests, or choose the 20B or 30B model and give each request a longer window. If you want both the big model and a long context, a second card, or an H200 with 141 GB listed on Hugging Face's Endpoints page at $5.00 an hour, buys headroom.

The decision is cheaper to make on rented hardware. A day on a Runpod Secure A100 at $1.79 an hour is $42.96 and tells you more than a spreadsheet.

Migrating from Ollama to vLLM

If you started on Ollama, the move is small because both speak an OpenAI-style API.

1. Note the model each team member uses and its context length.
2. Find the Hugging Face repository for the same model. Ollama tags and Hugging Face names differ, and vLLM loads from Hugging Face by default.
3. Start vllm serve with that model on a GPU host, with an API key.
4. Change the clients' base URL from port 11434 to port 8000 and add the key. Agents that use Ollama's own provider setting may need to switch to an OpenAI-compatible provider. The Ollama docs show this pattern for OpenCode, with @ai-sdk/openai-compatible and a baseURL.
5. Check tool calling with the right parser before everyone switches.
6. Run both servers side by side for a week, then retire the old one.

Differences to expect: vLLM serves one model per process, so each extra model is another process or server. It also does not unload idle models the way Ollama's keep-alive does. The model stays in memory until you stop it.

Observability

A server nobody watches fails quietly. What to watch, in rough order of value:

• GPU memory. If it is near full at idle, the cache budget is too small.
• Queue length and waiting requests. Growth at peak hours means you need more capacity.
• Time to first token. This is what developers feel in the editor.
• Error rate by key. A misconfigured client often shows up as one key with many errors.
• Restarts. A crash loop after a driver update is common enough to alert on.

vLLM exposes an HTTP server you can put health checks against, and your proxy gives you per-key counts. Which metrics endpoints and names the current release exposes is worth checking in the docs for your version.

A deployment checklist

• Linux host with a supported driver, and a GPU that holds the model with cache to spare.
• vLLM installed from a pinned version or container tag.
• --api-key set, proxy and TLS in front, raw port firewalled.
• --enable-auto-tool-choice and the right --tool-call-parser if agents will use it.
• Context length and parallelism capped to what memory allows.
• A health check and a restart policy.
• Per-key rate limits at the proxy.
• A fallback: a second node, a hosted API with a data agreement, or a plan to wait.
• A named person who owns upgrades.

Securing a vLLM server

The server checks a key if you pass --api-key, and that is all the built-in access control the docs describe. Treat that as a minimum:

1. Always set --api-key when the port is reachable from anywhere but localhost.
2. Put a reverse proxy in front with TLS, and firewall port 8000.
3. Use a different key per user or per application at the proxy, so a leaked key can be revoked alone.
4. Log request metadata, not full prompts, unless audit rules require them.
5. Keep the Hugging Face token for model downloads out of the serving environment where you can.

Operating it

Pin the version. vLLM is moving quickly, and v0.32.0 is the release we saw on 18 October 2026. Upgrade on a schedule, in a staging copy first, and run a handful of real prompts through each model, including a tool-calling one. Watch GPU memory, request queue length and time to first token. If the queue grows at peak hours, add a second replica behind the proxy, or move to a bigger card. Keep the previous container tag so rollback is quick.

When vLLM is the wrong choice

• One user on a laptop. Ollama or LM Studio is simpler.
• A Windows-only team without WSL. vLLM does not run natively there.
• You want many different models switched on demand. vLLM serves one model per process by default, so it suits a fixed model better than a library.
• No one to run it. A hosted API removes the operations burden.

Our take

Pick vLLM when the question is "how many people can this GPU serve". Its documentation is clear about the parts that matter to a team: an OpenAI-compatible API, API keys, tool calling and multi-GPU settings. Start on one rented 80 GB card, measure real concurrency, and only then decide whether to buy. Keep it behind a proxy. These judgements come from the docs and pricing pages in the frontmatter, and we did not benchmark it.

Related pages

• Private AI hub and Self-hosted AI
• How to self-host an LLM
• Private AI for teams
• Hosting hub