Run an LLM Locally: Hardware, Tools and Setup

To run an LLM locally, install a runtime such as Ollama or LM Studio, download an open-weight model that fits your memory, and chat with it on your own machine. A 20B-class model needs about 16 GB of memory according to its Ollama page. Nothing leaves your computer. We checked the tools and sizes on 31 October 2026.

Pick your path

Not sure which tool? Ollama vs LM Studio breaks it down. The rest of this page explains what is happening under the command so that you can fix problems yourself.

What "running locally" means

A language model is a large file of numbers called weights. A runtime loads those weights into memory and does the arithmetic whenever you send a prompt. "Local" means both happen on your hardware. There are three consequences.

Memory decides what runs. The whole model, plus working memory for the conversation, has to sit in fast memory. On a PC that is GPU VRAM. On an Apple silicon Mac it is the shared unified memory. If it does not fit, the runtime moves part to the CPU and the speed falls.

Speed depends on memory bandwidth and the chip. A model that fits entirely on the GPU answers far faster than one split between GPU and CPU. Ollama's ollama ps command shows the split, for example 48 percent CPU and 52 percent GPU.

Privacy is the default. Ollama's FAQ says that when you run locally it does not see your prompts or data. That is the main reason people do this, along with offline use and no per-token bill.

Step 1: choose a tool

Ollama

Ollama is an MIT-licensed runtime and model manager. It runs on macOS, Windows and Linux, and also in Docker. Install it from ollama.com, then run a model:

ollama run gpt-oss:20b

The first run downloads the model, which is 14 GB for gpt-oss 20B on its Ollama page, then opens a chat. Ollama serves an HTTP API on port 11434, with an OpenAI-compatible endpoint under /v1/.

LM Studio

LM Studio is a desktop app. You search for a model inside it, download it and chat in a window. Its system requirements page says: on a Mac you need Apple silicon and macOS 14 or newer, with 16 GB of RAM recommended and 8 GB workable for small models. On Windows it supports x64 with AVX2, and ARM Snapdragon X Elite, with 16 GB of RAM and at least 4 GB of dedicated VRAM recommended. On Linux it supports x64 and ARM64. Intel Macs are not supported.

llama.cpp

llama.cpp is the C and C++ engine that many apps build on. It is MIT licensed. Its README lists Apple silicon as a first-class target and backends for CUDA, HIP, Vulkan, SYCL and Metal. You run a model straight from Hugging Face:

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

That example from the README starts a server for a tiny model, a good smoke test before you try a bigger one. For the trade-off against Ollama, see llama.cpp vs Ollama.

Step 2: choose a model that fits

Two numbers matter: the file size on disk and the context window. The Ollama library shows both. Examples from model pages we read on 31 October 2026:

Rule of thumb, which is arithmetic and not a figure from a vendor: weights need about parameters times bytes per parameter. At 16-bit that is 2 bytes, at 8-bit 1 byte, at 4-bit about half a byte. Then leave 20 to 30 percent free for the conversation cache.

Quantisation is the compression that makes 4-bit possible. It saves memory and costs some quality. For most people on a laptop, the 4-bit build of the biggest model that fits is the right default. Our GPU requirements for LLMs page tabulates the sizes.

Step 3: set the context length

This is the setting that surprises beginners. Per Ollama's docs, the default context length depends on VRAM: 4k tokens under 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. A small window means the model forgets the start of a long document. Raise it as far as your memory allows.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

A longer context uses more memory. If speed drops after you raise it, run ollama ps and check whether the model has spilled onto the CPU.

Step 4: call it from code

Because the API is OpenAI-compatible, existing client libraries work. Point the base URL at your machine:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
    model="gpt-oss:20b",
    messages=[{"role": "user", "content": "Summarise this in two lines: ..."}],
)
print(reply.choices[0].message.content)

The key is a placeholder. Ollama does not check it locally. For the longer version with Docker, keep-alive and parallel requests, read how to self-host an LLM.

What a laptop can and cannot do

We make no speed claims, because speeds vary a lot by chip and are not in vendor docs. Here is what the docs support.

Can do: private summaries and drafts, offline chat, simple code questions, local search over your notes with an embedding model, and prototype apps that need an LLM without an API bill.

Struggles with: very long documents on small memory, agent loops that read many files, and anything where a frontier model's extra capability matters. For coding agents, Ollama's docs say Claude Code, Codex and OpenCode want a context of at least 64k, and Cline at least 32K. That is more memory than the default settings give you.

Hardware by platform

Nvidia GPUs on Windows and Linux

Ollama's hardware page lists support for Nvidia cards with compute capability 5.0 and above and driver version 550 or newer. Cards from 5.0 through 6.2 need driver 570 or newer. The list covers GeForce RTX 50, 40, 30 and 20 series cards, the professional RTX and A-series cards, L4, L40, A100, H100 and H200, and the GB10 in the DGX Spark. If you have several cards, CUDA_VISIBLE_DEVICES limits Ollama to a subset, and an invalid ID forces CPU use.

VRAM is the number to look at when you shop. A 24 GB card holds the 14 GB gpt-oss 20B build with room for context. A card with 8 GB holds small models only. A second identical card helps vLLM and llama.cpp split a larger model, though you should check each runtime's multi-GPU notes before buying.

AMD GPUs

Ollama supports AMD Radeon cards through ROCm on Linux, which requires the ROCm v7 driver, and adds more cards through Vulkan. Support depends on the exact card and OS, so read the current list before you buy. llama.cpp reaches AMD through HIP and Vulkan, and vLLM supports ROCm 6.3 or above per its install docs.

Apple silicon

Ollama uses the Metal API for Apple GPUs. LM Studio requires Apple silicon and macOS 14 or newer, and says it runs MLX and llama.cpp under the hood. The llama.cpp README calls Apple silicon a first-class citizen. The special advantage of a Mac is unified memory: the GPU can use most of the machine's RAM, so a Mac with a lot of memory can load models that a PC graphics card with the same VRAM cannot. We cannot give you tokens-per-second figures from vendor docs, so treat any speed number you see elsewhere as that author's measurement on that machine.

CPU only

Every runtime here can fall back to the CPU. It works for small models and testing. For anything you will wait on, the response time is the cost. If you must stay on CPU, pick the smallest model that does the job and keep the context short.

Match the model to the job

The right model depends on the work, not on a leaderboard.

A coding model is trained for code and tool use, so it often beats a general model of the same size on that job. The Qwen3-Coder page describes it as built for agentic and coding tasks with 256K native context. That description is the vendor's own, and we have not tested it.

Local search and notes

A common first project is asking questions about your own files. The pieces are an embedding model that turns text into numbers, a place to store them, and a chat model that answers using the best matches. Ollama serves embedding models through the same API as chat models, so one install covers both. The privacy upside is clear: your notes are never uploaded. The catch is that quality depends on how you split the text and how many matches you pass in, which is a design job and not a setting. Our explainer on MCP vs RAG covers when retrieval is the right tool.

From laptop to shared server

When two or three colleagues want the same model, you have outgrown the laptop. The options, in order of effort:

1. Share the Ollama box. Bind it to the network behind a proxy. Raise OLLAMA_NUM_PARALLEL from its default of 1 knowing that memory needs scale with it.
2. Move to vLLM. It is built for many concurrent requests. See vLLM and Ollama vs vLLM.
3. Rent a GPU. Pay by the hour and keep control of the software. See Runpod and Lambda Cloud.
4. Buy a server. See on-premise AI.

When local is the wrong answer

• You need the best answer, not a private one. A hosted frontier model will usually win on hard reasoning.
• Many people need it at once. One laptop does not serve a team. See private AI for teams.
• The machine is old. Ollama needs an Nvidia card with compute capability 5.0 or higher and a recent driver to use the GPU. Below that you are on the CPU.
• You only need it twice a month. A hosted API costs cents. A rented GPU by the hour is the middle path: see hosting and the hosting compare.

Cost of running locally

The software is free. The costs are the hardware you already have, electricity, and your time. If you buy a machine for this, the break-even against API spend is a real calculation: our worked example in the self-host guide uses example figures of $400 a month in API spend against a $3,000 workstation, and lands near 22 months. Plug in your own numbers.

If your laptop is too small, rent for an afternoon. Example, with volumes we chose: a Runpod RTX 4090 at $0.89 an hour on the Secure tier, used for 10 hours in a month, costs $8.90. Prices checked 31 October 2026. Some links go through /go/ and may earn us a fee; it never changes our order.

Privacy checklist for a local setup

1. Keep the server on localhost unless you must share it. Ollama binds to 127.0.0.1 by default.
2. If you share it, put a proxy with authentication in front, and firewall the raw port.
3. Turn off cloud features in Ollama if you want local-only. Its FAQ says that disables cloud models and web search.
4. Download weights from the publisher's page.
5. Know where your chat history is stored. A desktop app keeps it on disk, and your backup tool may copy it elsewhere.

Troubleshooting

Our take

Running a model locally is the lowest-risk way to learn what open models can do. It costs nothing, nothing leaves your machine and you can undo it by deleting a folder. Start with Ollama if you are comfortable in a terminal and LM Studio if you want a window. Aim for a model that leaves a quarter of your memory free. If it works for your task, you have a private tool. If it does not, you have learned that cheaply, and the next step is a larger machine, a rented GPU, or a hosted API.

We have not benchmarked these tools against each other for this page. Our statements come from the vendors' docs and model pages listed in the frontmatter.

Related pages

• Private AI hub and the self-hosted AI overview
• Ollama
• Local coding agent with Ollama
• Hosting hub for when a laptop is not enough