Self-hosted runtime

vLLM

vLLM is a high-throughput, memory-efficient serving engine for large language models, run as a server you host yourself.

Website (opens in a new tab)Source (opens in a new tab)

Checked 2026-10-31

vLLM is a high-throughput, memory-efficient serving engine for large language models, run as a server you host yourself. Starting it is one command, vllm serve with a model name, and it then serves the OpenAI Chat Completions API at /v1/chat/completions. The docs note that endpoint applies only to text generation models with a chat template, and you can pass one with a flag if a model lacks it. The installation page lists NVIDIA CUDA, AMD ROCm, Intel XPU and Apple Silicon through vLLM-Metal for GPUs, and Intel and AMD x86, ARM AArch64, Apple silicon and IBM Z for CPUs. It is released under the Apache 2.0 license and has no fee. We would suggest it to teams serving open models to many users who care about throughput and GPU memory. It is less suited to a casual desktop user, because it has no app and expects you to manage Python, drivers and hardware. The pages we read do not say which operating systems are supported. We have not run it ourselves for this note.

Mentioned here

Questions

How much does vLLM cost?

Free and open source (Apache 2.0); you pay only for your own hardware (checked 31 October 2026).

Is vLLM OpenAI compatible?

Yes, it offers an OpenAI-compatible API, so most agents and SDKs can point at it by changing the base URL.

Can I self-host vLLM?

Yes, it can run on your own servers.

Reviews

Loading reviews…

Write a review

Your rating

Up to 1000 characters. Stars alone are enough.

Similar

  • Text Generation Inference

    Text Generation Inference, or TGI, is Hugging Face's open server for running large language models on your own GPUs.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • llama.cpp

    llama.cpp is a plain C and C++ engine for running language and vision models with little setup.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • LM Studio

    LM Studio is a desktop app for downloading and running open-weight models on your own computer.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • LocalAI

    LocalAI is an open-source runtime that runs AI models on your own hardware.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • Jan

    Jan is a desktop app from Menlo Research for running open models on your own machine.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable