Self-hosted runtime

llama.cpp

llama.cpp is a plain C and C++ engine for running language and vision models with little setup.

Website (opens in a new tab)Source (opens in a new tab)

Checked 2026-10-31

llama.cpp is a plain C and C++ engine for running language and vision models with little setup. The README says Apple silicon is a first-class target, with optimized paths for AVX on x86, plus CUDA for NVIDIA GPUs, HIP for AMD, Vulkan, SYCL, and CPU and GPU hybrid inference for models larger than your video memory. It supports integer quantization from 1.5-bit to 8-bit to cut memory use. Install options are an install script for macOS, Linux and Windows, Docker, prebuilt binaries, or a build from source. Once installed, the llama serve command launches an OpenAI-compatible API server, and a built-in web UI is shown in the README. It is released under the MIT license and costs nothing. We would suggest it to people who want control over model files, flags and hardware, and to anyone embedding inference in their own software. The main limit is that it is a command-line tool, so you choose models, quantization levels and settings yourself, and the README does not give a single recommended configuration for each machine. We have not run it ourselves for this note.

Mentioned here

Questions

How much does llama.cpp cost?

Free and open source (MIT); you pay only for your own hardware (checked 31 October 2026).

Is llama.cpp OpenAI compatible?

Yes, it offers an OpenAI-compatible API, so most agents and SDKs can point at it by changing the base URL.

Can I self-host llama.cpp?

Yes, it can run on your own servers.

Reviews

Loading reviews…

Write a review

Your rating

Up to 1000 characters. Stars alone are enough.

Similar

  • Text Generation Inference

    Text Generation Inference, or TGI, is Hugging Face's open server for running large language models on your own GPUs.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • vLLM

    vLLM is a high-throughput, memory-efficient serving engine for large language models, run as a server you host yourself.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • Jan

    Jan is a desktop app from Menlo Research for running open models on your own machine.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • SGLang

    SGLang is a serving framework for large language and multimodal models, aimed at low latency and high throughput from a single GPU up to large clusters.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable