Self Hosted AI Coding Assistant: Options and Design

A self-hosted AI coding assistant is three parts: a model server you run, an editor or terminal tool that talks to it, and a gate in front for access control. Ollama or vLLM serves the model. Continue, Cline, OpenCode or Tabby are the client side. Your code stays inside your network if every part runs there. We checked the options on 31 October 2026.

The options

Tabby's repository describes itself as self-contained, with no DBMS or cloud service needed, an OpenAPI interface, and support for consumer-grade GPUs. Its last release was in January 2026, so check recent activity before you commit a team to it. The repository is not archived and had a push on 30 June 2026. Its LICENSE puts content outside the ee/ directory under Apache 2.0 and content under ee/ under a separate ee/LICENSE, so read that file before you use the enterprise features.

Profiles for the agents are at Cline, OpenCode, OpenHands, Aider and Goose.

What an assistant actually has to do

"Coding assistant" covers three jobs, and they place different demands on the server.

Autocomplete. Suggestions appear as you type. Delay is everything: the answer has to arrive in a fraction of a second, so it needs a small, fast model and a server close to the developer. The prompt is short and the output shorter.

Chat. You ask a question about code and get an answer. Latency matters less. A mid-size model works, and a larger context window lets you paste more files.

Agent mode. The tool reads files, edits them and runs commands, in loops. It needs a model with tool calling and a large context, and it uses many requests per task. Ollama's docs ask for at least 64k of context for Claude Code, Codex and OpenCode, and 32K for Cline.

A common mistake is to size a server for chat and then switch on agent mode. Agent loops use many times more tokens and hold the model busy for minutes. Plan for the heaviest job you intend to allow. Check the model card of your completion model for fill-in-the-middle support before you choose it.

Architecture for a team

Developer editors and terminals
        |  HTTPS, API key per user
        v
   Reverse proxy (TLS, auth, rate limits, logs)
        |
        v
   Model server: vLLM or Ollama
        |
        v
   GPU(s) in your rack or a rented host

Each layer has a job.

The model server. Ollama is the quickest start and runs on one machine with an OpenAI-compatible endpoint on port 11434. vLLM is built for many concurrent requests, serves one model per process by default, and speaks the OpenAI protocol on port 8000. For tool calling on vLLM you pass --enable-auto-tool-choice and a --tool-call-parser suited to the model. The vLLM docs list parsers for several model families, including hermes and openai for OpenAI's open models. See Ollama vs vLLM.

The proxy. Ollama has no login. vLLM can check an API key with --api-key, and you can pass several for rotation. A reverse proxy adds TLS and per-user keys, so you can see who is using the GPU and revoke one person without breaking the others.

The clients. Each developer points the tool at your base URL. That is a config line for Continue, a provider setting for Cline, and an opencode.json entry for OpenCode, as in our local coding agent guide. Ollama's docs also list integrations for VS Code, JetBrains, Zed and Xcode.

Models and memory

Pick the model by the heaviest job. From Ollama's library pages:

Cache scales with context and parallel requests. For five developers in agent mode on a 120B model on one 80 GB card, you will hit memory limits before you hit compute limits. Our GPU requirements for LLMs page works through that sum.

What it costs

Software here is free. The server is the cost.

Rented GPU (example volumes we chose). An 80 GB A100 on Runpod Secure is $1.79 an hour, prices checked 31 October 2026. For a team that uses it 10 hours a day on 22 working days, that is 220 hours, about $394 a month. Left on 24/7 in a 30-day month it is about $1,289. A Runpod Community H100 SXM is $2.69 and Secure is $3.99, and Lambda lists a single H100 SXM at $4.29. Some links go through /go/ and may earn us a fee; it never changes our order.

Hosted open-model API. If you do not need the server to be yours, a hosted API charges per token. Groq lists gpt-oss-120b at $0.15 per million input tokens and $0.60 per million output tokens. Cloudflare Workers AI lists it at $0.35 and $0.75. For a team of five using, say, 40 million input and 8 million output tokens a month between them (an example, not a measurement), Groq's rates give $6.00 plus $4.80, about $10.80. That is far below the cost of a rented GPU, which is why this is a privacy decision before it is a cost decision.

Owned hardware. A workstation with a 48 GB or 80 GB GPU is a capital purchase. Compare it with rental using the break-even sum in our self-host guide.

For any host, check the hosting hub and the interactive hosting compare.

Privacy and control

What you control: where the prompts go, what is logged, how long it is kept, and who can reach it. What you must still check:

• Telemetry. Each extension and agent has its own usage reporting. Read the docs and disable what you must. Cline's docs say telemetry is on by default in its extensions, desktop app and CLI, with an opt-out in settings or "telemetryOptOut": true in the CLI's global settings. We did not confirm the defaults for Continue, OpenCode or Tabby.
• Extra services. Web search, web fetch and cloud model options in some tools send data out. Ollama's agent integrations route web search through Ollama's service.
• Index data. Some assistants build a search index of your repository. Know where it is stored and who can read it.
• Logs. A proxy that logs prompts for audit is also a store of your source code. Protect it like the repo.
• Model provenance. Download weights from the publisher, and record the version.

Building the server side, step by step

This is the shared-server path, using commands from the vLLM docs.

1. Install vLLM on a Linux GPU host. The quickstart uses uv:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

Linux is required. vLLM does not support Windows natively, so use WSL if you must.

2. Serve a model with tool calling and a key. Agent mode needs the tool-call flags. The parser must match the model family, so read the tool calling page for yours:

vllm serve <model-id> \
  --api-key YOUR_LONG_RANDOM_KEY \
  --enable-auto-tool-choice \
  --tool-call-parser <parser-for-your-model>

The docs show llama3_json for a Llama example, and list hermes and openai as parsers for other families. The server listens on port 8000 by default and speaks the OpenAI protocol. A wrong parser is the most common reason tool calls fail on a self-hosted agent, so test with a tiny agent task before you roll it out.

3. Check it from a client.

curl http://localhost:8000/v1/models -H "Authorization: Bearer YOUR_LONG_RANDOM_KEY"

4. Put a proxy in front. Terminate TLS, forward to port 8000, and keep the raw port firewalled. Issue one key per developer at the proxy if your proxy supports it, so that you can revoke without restarting vLLM.

5. Point the clients. The base URL is https://your-host/v1, the key is the developer's own, and the model name is what /v1/models reports.

If you prefer Ollama for the first version, the same shape applies with port 11434, OLLAMA_HOST, and a proxy for authentication. See how to self-host an LLM for those steps.

Choosing the client

Mixing is fine. Many teams run an autocomplete and chat tool in the editor and a terminal agent for larger tasks, all pointing at the same server. For the product comparison of the agents themselves, see AI coding agents and Cline vs Roo Code.

Governance a team needs

A shared model server becomes shared infrastructure, and a few rules save arguments later.

Access. Give each developer a key and a way to lose it. Tie keys to your identity provider if the proxy supports it. Remove access when someone leaves.

Limits. One person's agent loop can starve everyone else. Set per-key rate limits at the proxy and cap context length at the server. Ollama's OLLAMA_NUM_PARALLEL and context settings are the server-side levers.

Logging. Decide what you log. Request metadata, such as who called, when and how many tokens, is enough for capacity planning. Full prompt logging stores your source code in a second place, so only turn it on if audit rules require it, and protect the log accordingly.

Policy. Write down what may go into prompts. Secrets, customer data and licensed third-party code are common exclusions, and a self-hosted server does not remove the need for the rule.

Changes. Treat a model change like a dependency upgrade. Announce it, test it on a sample of real tasks, and keep the previous version available for rollback.

What can go wrong

The server runs out of memory under load. Symptom: it works for one developer and fails for five. Cause: cache grows with parallel requests and context. Fix: cap context, cap parallelism, or add capacity.

Agent edits are wrong. Symptom: the assistant changes unrelated files. Cause: a model too small for the task or a context too short to hold the project. Fix: a bigger model for agent mode, or narrower tasks.

Tool calls break. Symptom: the agent prints JSON instead of acting. Cause: the tool parser does not match the model. Fix: check the vLLM tool calling page for the right parser.

Developers route around it. Symptom: people paste code into a public chatbot anyway. Cause: the private assistant is slower or worse. Fix: measure and improve it, and offer a fast path for the hard cases, with a policy on what may go there.

The tool project goes quiet. Symptom: no releases for months. Fix: prefer clients with recent activity and a permissive licence, and keep the server API-compatible so you can swap clients.

Limits to plan for

• Quality. Local models are not frontier models. Expect them to struggle on long tasks that touch many files. We have no verified benchmark to put a number on it.
• Operations. A shared GPU server needs patching, monitoring and a person who owns it.
• Upgrades. Runtimes and models move fast. Pin versions and upgrade on a schedule.
• One model per vLLM process. If you want a fast completion model and a big chat model, that means two processes or two servers.

Picking by situation

Rollout plan

1. Week one. One developer runs Ollama locally with Continue or OpenCode on a real task. Note where it fails.
2. Week two. Move the model to a shared box. Add the proxy and keys. Invite three colleagues.
3. Week three. Measure: requests per hour, peak concurrency, memory use from ollama ps or vLLM logs.
4. Week four. Decide whether to scale, change models, or fall back to a hosted option for hard tasks.

At each step, write down what you learned. The notes are your sizing data for the next purchase.

Our take

The cleanest self-hosted setup is boring: one model server, one open-source client, one proxy, and clear ownership. Choose tools by how active the repository is and how clear the licence is, and be wary of any assistant that has not shipped in months. Keep the privacy claim narrow. "Prompts stay in our network" is true only if every tool, plugin and search feature in the chain stays in the network too. Our conclusions come from documentation and repository data, not from a test of output quality.

Related pages

• Private AI hub and Private AI for teams
• Self-hosted AI
• AI coding agents
• Hosting hub