GPU Requirements for LLMs: VRAM by Model Size

A language model needs enough fast memory for its weights plus a conversation cache. At 4-bit, weights take roughly half a byte per parameter, so a 7B model needs about 4 to 5 GB, a 32B model about 20 GB and a 70B model about 43 GB. Add 20 to 30 percent for the cache. Download sizes below come from Ollama model pages checked on 31 October 2026.

The quick table

Sizes are the default tag download size shown on each Ollama model page. They are what the weights take on disk, and roughly what they take in memory before the cache.

Context windows listed on the same pages run from 40K on some qwen3 tags to 128K for gpt-oss, gemma3 and Llama 3.3, and 256K for Qwen3-Coder and some qwen3 tags. A model's maximum context is not the same as what your card can hold. The next sections show why.

The formula

Weights are simple. Memory for weights is the parameter count times bytes per parameter.

These are our arithmetic, not vendor figures, and real files run larger because of overhead and because 4-bit schemes keep some layers at higher precision. The listed Llama 3.3 70B at 43 GB against the 35 GB estimate shows the gap, around 20 percent. Use the listed size when a model page gives one.

The special case is gpt-oss. OpenAI post-trained it with the mixture-of-experts weights in MXFP4, about 4.25 bits per parameter, and those weights are over 90 percent of the parameters. That is why the 120B model fits an 80 GB GPU and the 20B model runs in 16 GB, per the Ollama page.

The cache you forget

While a model reads and writes, it stores keys and values for every token in the conversation. This key-value cache grows with context length, and it is multiplied by the number of requests handled at the same time. Ollama's FAQ states that the required memory scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH.

What that means in practice:

• A 14 GB model on a 16 GB card works for short chats and fails for 64k coding sessions.
• Four users on the same model need more cache than one user, even with the same weights.
• Doubling context roughly doubles cache. The exact size depends on the model's architecture, which model cards describe.

Ollama sets a default context from your memory: 4k tokens under 24 GiB of VRAM, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. Coding agents need more than 4k. Ollama advises 64k or higher for Claude Code, Codex and OpenCode, and 32K or more for Cline.

Quantisation: how much quality do you give up

Quantisation stores each weight in fewer bits. It cuts memory and speeds up decoding, because less data moves. The cost is some quality loss, which grows as bits shrink.

A useful rule is to prefer a larger model at 4-bit over a smaller one at 8-bit when both fit, then check on your own prompts. We cannot cite a single primary source for the exact quality loss at each level, because it varies by model and task.

What cards offer

VRAM of common cards, taken from rental and hardware pages we checked on 31 October 2026 unless noted:

On Apple silicon the number that matters is unified memory, which the GPU shares with the system. A Mac with 64 GB can hold the 43 GB Llama 3.3 70B download. LM Studio recommends 16 GB for Macs and says 8 GB works for small models.

Software support by vendor

The card must be supported by the runtime you pick.

• Ollama: Nvidia compute capability 5.0 and up with driver 550 or newer, AMD through ROCm v7 on Linux and Vulkan, and Apple through Metal.
• vLLM: Linux only. NVIDIA CUDA, AMD ROCm 6.3 or above, Intel XPU, and Apple silicon through a community plugin, vLLM-Metal. No native Windows.
• llama.cpp: CUDA, HIP for AMD, Metal, Vulkan, SYCL and more, per its README.

See Ollama, vLLM and Ollama vs vLLM.

When one card is not enough

There are two ways to spread a model across cards. Pipeline splitting puts different layers on different GPUs, and tensor parallelism splits each layer. vLLM supports multi-GPU serving, and llama.cpp and Ollama can split layers across devices. Multi-GPU helps capacity more than speed, because cards talk over PCIe or NVLink. Data-centre cards with fast links, such as the H100 SXM variants, are why rental prices for SXM are higher than for PCIe.

If your model needs more than one 80 GB card, ask whether a smaller model or a harder quantisation would do the job. A 120B-class model on one H100 is a very different bill from a 405B model on four.

What renting costs

Prices checked on 31 October 2026. Some links on our site go through /go/ and may earn us a fee; it never changes our order.

Notice that the cheapest way to hold 80 GB is a Runpod A100 at $1.79 an hour on the Secure tier, and that the H100 costs more for speed, not memory. For gpt-oss 120B, which fits in 80 GB, the A100 is the capacity fit and the H100 is the throughput fit.

An example with volumes we chose: a team uses an 80 GB A100 for 6 hours a day, 22 days a month. That is 132 hours at $1.79, about $236 a month. The same card left on 24/7 for a 30-day month is 720 hours, about $1,289. See Runpod, Lambda Cloud, Runpod vs Lambda and the interactive hosting compare.

Three worked sizing examples

These use listed sizes and our own arithmetic. The numbers are examples to show the method.

Example one: a solo developer, coding agent, 64k context. The agent needs a model with tool calling and a long window. Take Qwen3-Coder 30B, listed at 19 GB. Add about 25 percent for the cache at 64k, which gives roughly 24 GB. That is the edge of a 24 GB card, so the sensible plan is a 32 GB or 48 GB card, or a Mac with 48 GB or more of unified memory. If the budget is a 24 GB card, drop the context to 32k and check ollama ps.

Example two: five people share gpt-oss 120B. The weights are 65 GB. Five parallel requests multiply the cache. One 80 GB card leaves about 15 GB for cache, which supports modest context for a handful of users but not five long agent sessions at once. The choices are to cap context, to cap parallelism, or to add a second card. vLLM's batching helps throughput, though it does not remove the memory limit.

Example three: a quick experiment on a 70B model. Llama 3.3 70B is listed at 43 GB. You do not need to buy anything. An L40S on Runpod Secure has 48 GB for $1.09 an hour, so a four-hour test costs $4.36. If it fits your task, buy or rent long term. If not, you have lost the price of a coffee.

Capacity versus speed

Two different limits apply, and cards trade them differently.

Capacity is how much fits. It decides whether a model runs at all. An A100 80 GB and an H100 80 GB have the same capacity, so they hold the same model.

Speed depends on how fast memory feeds the chip and on how much compute the card has. Decoding one token at a time is mostly limited by memory bandwidth, because every token reads most of the weights. This is why 4-bit models decode faster than 16-bit ones, and why data-centre cards beat consumer cards of equal size. The vendor pages we read do not publish tokens per second for these cards, so we give none.

For a team, the useful number is requests served at an acceptable delay, and that depends on batching. This is the reason vLLM exists, and why we point shared servers there.

Fine-tuning needs more than inference

Everything above is for running a model. Training or fine-tuning needs memory for gradients and optimiser state as well, so the requirement is several times higher than inference for the same model. Techniques that train small adapter layers on a quantised base model cut that sharply. If fine-tuning is the goal, price it separately and consider renting by the hour, since training runs are short and bursty. Our hosting hub lists providers that rent GPUs and offer serverless options.

Apple, Nvidia or rented: a decision table

The break-even sum is the same one in our self-host guide: hardware cost divided by the monthly saving over renting, with your own time counted as a cost.

Choosing hardware by goal

A good habit is to pick the model first, find its listed size, then add 20 to 30 percent for cache and pick the card. Do not start from a card you already own and hope.

Checks before you buy

1. Read the model's page for its listed size, licence and context.
2. Add 20 to 30 percent for cache, more for long context or several users.
3. Confirm the runtime supports the card and driver.
4. Rent for a day first. An hour of A100 time at $1.79 is cheaper than a wrong purchase.
5. After install, run ollama ps and confirm the model sits fully on the GPU.

Our take

Memory capacity is the first filter and bandwidth the second. Most buying mistakes come from sizing for the weights and forgetting the cache, so we would plan for a quarter extra and test with the real context length. For one person, 16 to 24 GB covers a lot. For a team, one 80 GB card and a 120B-class model is the sweet spot that vendors document. Beyond that, renting and hosted APIs usually beat buying for most organisations. All figures come from the pages in the frontmatter, and we ran no benchmarks.

Related pages

• Private AI hub and hosting hub
• Run an LLM locally
• How to self-host an LLM
• Self-hosted AI