Self-hosted runtime

SGLang

SGLang is a serving framework for large language and multimodal models, aimed at low latency and high throughput from a single GPU up to large clusters.

Website (opens in a new tab)Source (opens in a new tab)

Checked 2026-10-31

SGLang is a serving framework for large language and multimodal models, aimed at low latency and high throughput from a single GPU up to large clusters. It is not an app; you run it as a server for your own models. The docs say it is compatible with the OpenAI API. The README lists NVIDIA GPUs from RTX 30 series to B200 and GB300, AMD Instinct MI300X and newer, Google TPU, and other accelerators including Ascend NPU. Install is by Docker image, pip or uv, or from source. The install guide says Linux with NVIDIA GPUs is the main target, Python 3.11 or higher is needed, and CUDA 13 is required, with CUDA 12 images and wheels no longer published after version 0.5.19. The code is under the Apache 2.0 license and there is no fee. We would suggest it to engineers who serve open models in production and want to tune throughput. It is a poor fit for someone who wants a point-and-click desktop tool, and the CUDA requirement means older drivers will need an upgrade. We have not run it ourselves for this note.

Mentioned here

vLLM
vLLM is a high-throughput, memory-efficient serving engine for large language models, run as a server you host yourself.
- Self-hosted runtime
- OpenAI compatible
- Self-hostable
Runpod
Runpod rents GPUs by the second, either as pods you control or as serverless endpoints.
- GPU cloud
- OpenAI compatible
Modal
Modal runs Python code on GPUs and bills by the second, with no servers to manage: you decorate functions and Modal builds the container and scales it.
- Serverless GPU
vLLM: Serving Open Models to a Team, Setup and Limits
Checked 2026-10-31
How to Self-Host an LLM
Checked 2026-10-31
GPU Requirements for LLMs: VRAM by Model Size
Checked 2026-10-31
Cloud vs Self-Hosted AI: A Practical Guide

Questions

How much does SGLang cost?

Free and open source (Apache 2.0); you pay only for your own hardware (checked 31 October 2026).

Is SGLang OpenAI compatible?

Yes, it offers an OpenAI-compatible API, so most agents and SDKs can point at it by changing the base URL.

Can I self-host SGLang?

Yes, it can run on your own servers.

Reviews

Loading reviews…

Similar

Text Generation Inference
Text Generation Inference, or TGI, is Hugging Face's open server for running large language models on your own GPUs.
- Self-hosted runtime
- OpenAI compatible
- Self-hostable
llama.cpp
llama.cpp is a plain C and C++ engine for running language and vision models with little setup.
- Self-hosted runtime
- OpenAI compatible
- Self-hostable
LM Studio
LM Studio is a desktop app for downloading and running open-weight models on your own computer.
- Self-hosted runtime
- OpenAI compatible
- Self-hostable
LocalAI
LocalAI is an open-source runtime that runs AI models on your own hardware.
- Self-hosted runtime
- OpenAI compatible
- Self-hostable
Jan
Jan is a desktop app from Menlo Research for running open models on your own machine.
- Self-hosted runtime
- OpenAI compatible
- Self-hostable