Self-hosted runtime

Text Generation Inference

Text Generation Inference, or TGI, is Hugging Face's open server for running large language models on your own GPUs.

Website (opens in a new tab)Source (opens in a new tab)

Checked 2026-11-05

Text Generation Inference, or TGI, is Hugging Face's open server for running large language models on your own GPUs. Start with its status. The docs open with a caution that TGI is in maintenance mode, taking only minor bug fixes, documentation improvements and lightweight maintenance, and the GitHub repository shows as archived and read-only since 21 March 2026. Hugging Face points new work to vLLM and SGLang, or to llama.cpp and MLX for local use. The code is Apache-2.0 licensed, so there is no licence fee. You pay for whatever hardware you run it on. The docs list continuous batching, tensor parallelism across GPUs, token streaming over server-sent events, Flash Attention and Paged Attention, bitsandbytes and GPT-Q quantization, and Prometheus metrics with OpenTelemetry tracing. The repository lists NVIDIA, AMD, Inferentia, Intel GPU, Gaudi and Google TPU support. We would use it to keep an existing deployment running, and start any new build on a maintained engine. Because you host it yourself, prompts stay on your own machines, and the docs make no data claims of their own. Facts here come from the docs home page and the GitHub repository page we checked on 5 November 2026.

Mentioned here

  • vLLM

    vLLM is a high-throughput, memory-efficient serving engine for large language models, run as a server you host yourself.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • SGLang

    SGLang is a serving framework for large language and multimodal models, aimed at low latency and high throughput from a single GPU up to large clusters.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • llama.cpp

    llama.cpp is a plain C and C++ engine for running language and vision models with little setup.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • Ollama

    Ollama runs open models on your own machine and also offers hosted cloud models.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • Hugging Face Inference

    Hugging Face Inference Endpoints deploys a model from the Hub onto dedicated, autoscaling hardware.

    • Model / inference API
    • OpenAI compatible

Questions

How much does Text Generation Inference cost?

It is free to use.

Is Text Generation Inference OpenAI compatible?

Yes, it offers an OpenAI-compatible API, so most agents and SDKs can point at it by changing the base URL.

Can I self-host Text Generation Inference?

Yes, it can run on your own servers.

Reviews

Loading reviews…

Write a review

Your rating

Up to 1000 characters. Stars alone are enough.

Similar

  • LM Studio

    LM Studio is a desktop app for downloading and running open-weight models on your own computer.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • LocalAI

    LocalAI is an open-source runtime that runs AI models on your own hardware.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable

  • Jan

    Jan is a desktop app from Menlo Research for running open models on your own machine.

    • Self-hosted runtime
    • OpenAI compatible
    • Self-hostable