LiteLLM: AI Gateway and Python SDK Explained

LiteLLM is an open source tool that lets you call 100+ LLM providers through one OpenAI-style API. It comes in two forms. The Python SDK is a library you import. The AI Gateway, also called the proxy, is a server you host, which apps in any language call by changing their base URL. The gateway adds virtual keys, budgets, spend tracking and fallbacks. The core is MIT licensed. We checked the repo and docs on 31 October 2026.

The name is one of the most searched in this space, and the answers are scattered. Here we cover what each form does, how to install them, how coding agents such as Claude Code and Codex connect, how costs get tracked, what the licence covers and where it stops.

Quick facts

The SDK and the gateway are two products

People mix these up, so it helps to separate them first. The README states the split in a table.

The docs index puts the choice plainly. Use the gateway when several apps, teams, users or coding agents share models and need keys, budgets and spend tracking, or when a client is not written in Python. Use the SDK when one Python app calls models directly.

You can also combine them. The SDK has a litellm_proxy/ model prefix for calling a gateway as if it were a provider.

Install the SDK

The README and docs show uv.

uv add litellm

A first call uses the <provider>/<model> naming pattern, and provider keys come from environment variables.

from litellm import completion

# Provider keys come from environment variables, for example OPENAI_API_KEY or ANTHROPIC_API_KEY
response = completion(
    model="openai/gpt-6.1-sol",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

Every provider returns the OpenAI Chat Completions shape, so code that parses one response parses them all. Streaming uses stream=True, and async uses await litellm.acompletion(...).

There is also litellm-core, a separate distribution with the same import litellm API and fewer mandatory dependencies, and without the bundled dashboard, CLI entry points or optional extras. The docs warn that litellm and litellm-core own overlapping Python files, so install one per environment, never both. Use litellm when you want the proxy, the CLI or the extras.

Install the gateway

There are four documented routes.

One command. The quickstart script installs and starts the gateway, a Postgres database and the Admin UI with Docker Compose.

curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.sh | sh

On Windows PowerShell:

irm https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.ps1 | iex

It needs Docker, Podman or Rancher Desktop with Compose v2. The result is the gateway and Admin UI on http://localhost:4000 (the next free port if 4000 is taken), listening on 127.0.0.1 only. Files go to ~/litellm-gateway. The script generates a master key, a salt key and a database password into a .env file with mode 600, and does not print them. The docs say to keep that file: a new salt key makes stored provider keys unreadable. Read the script before you pipe it into a shell. The docs link both scripts for that reason.

Compose by hand. The docs show downloading docker-compose.quickstart.yml, writing two generated keys to .env, and running docker compose -f docker-compose.quickstart.yml up -d. You get the gateway on port 4000 and Postgres for models, keys and spend logs.

One container, no database. You supply a config.yaml. This path is small but limited, as the limits section below explains.

Production. LiteLLM documents Helm for Kubernetes, plus Terraform modules for AWS (ECS Fargate, Aurora, ElastiCache, ALB) and GCP (Cloud Run, Cloud SQL, Memorystore). The images are docker.litellm.ai/berriai/litellm and ghcr.io/berriai/litellm, signed with cosign. The README says images with the -stable tag go through 12-hour load tests before publishing.

If you only want to try the CLI, the README gives uv tool install 'litellm[proxy]' followed by litellm --model gpt-4o.

First run in the Admin UI

After the script finishes, the docs walk you through six steps:

1. Open http://localhost:4000/ui. The username is admin, and the password is the LITELLM_MASTER_KEY value in your .env.
2. Go to Models + Endpoints, then Add Model. Pick a provider, choose models and paste the provider key.
3. Click Test Connect, then Add Model. The model appears with its pricing mapped.
4. Use the Playground to send a message.
5. Under Virtual Keys, create a key. It is shown once.
6. Call the gateway from your app with that key.

Treat the master key like a root password. The docs say anyone who holds it has full admin access, and that it also works as the default Admin UI password.

Providers and the OpenAI-compatible endpoint

The README's provider table covers well over a hundred names. A short sample: OpenAI, Anthropic, Azure, AWS Bedrock and SageMaker, Google Vertex AI and Gemini, Mistral, Cohere, Groq, Together AI, Fireworks AI, DeepSeek, xAI, OpenRouter, Ollama, vLLM, LM Studio, Hugging Face and NVIDIA NIM. A custom OpenAI-compatible server is also supported, which covers self-hosted models.

The table also shows which endpoints each provider supports. Not every provider supports every endpoint, so check the row before you assume /embeddings or /audio will work for yours. The docs note that a missing provider can be requested through a feature request.

The gateway is OpenAI-compatible and Anthropic-compatible, per its docs. An app keeps its existing SDK and changes the base URL.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-<your-virtual-key>")
response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

Here the model is a name you added in the Admin UI, not necessarily a provider's own model id. That indirection is the useful part. You can point fast at one model today and another next month, and no client changes.

Routing, fallbacks and auto routing

The Router handles retries, cooldowns, load balancing and fallbacks across deployments, in the SDK and on the gateway. A fallback is declared as a map from one model name to the names to try next.

fallbacks=[{"gpt-6-luna": ["gpt-6.1-sol"]}]

The README also shows an Auto Router, which picks a model for each prompt. Its example sends simple prompts to a fast model and complex or reasoning prompts to a capable one, using a heuristic classifier.

Other things the gateway does

• MCP gateway. Add MCP servers to the gateway and call their tools through /chat/completions, or point Cursor at the gateway's /mcp/ URL. If you are new to the protocol, our MCP vs API comparison is a starting point.
• A2A. The gateway can invoke A2A agents, with LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore and Pydantic AI named as supported providers.
• Guardrails. The open source framework includes custom guardrails and Presidio for PII masking.
• Caching and logging. Response caching, plus per-project logging and observability callbacks such as Langfuse and MLflow.

Using LiteLLM with coding agents

This is the main reason many readers land here. LiteLLM documents client setups for Claude Code, Codex, Cursor, OpenCode and others.

Claude Code

Claude Code talks to the Anthropic Messages API. LiteLLM serves that format at /v1/messages, so two environment variables point Claude Code at the gateway, and per the docs it then works with any model in your config, not just Anthropic's.

export LITELLM_API_KEY="sk-<your-virtual-key>"
export ANTHROPIC_BASE_URL="http://localhost:4000"
export ANTHROPIC_AUTH_TOKEN="$LITELLM_API_KEY"
export ANTHROPIC_MODEL="claude-sonnet-5"
claude

Details that save time:

• ANTHROPIC_MODEL must be a model_name that exists on your gateway.
• If ANTHROPIC_API_KEY is also set in your shell, unset it for the session, so Claude Code does not send it instead of the virtual key.
• The /model picker inside a session still lists Anthropic's own ids. Choosing one fails with Invalid model name passed in unless your config defines that exact name. Either add those names to model_list or keep switching through ANTHROPIC_MODEL.
• To keep the exports, put them in your shell profile or the env block of Claude Code's settings.json.

To check, send a prompt, then look in the Admin UI under Logs or Usage, where the request should be attributed to your virtual key and model.

Claude Code can also use the gateway's MCP tools. The docs give a claude mcp add --transport http command with the URL <gateway>/<server_name>/mcp and an x-litellm-api-key header. Our Claude Code listing and first-run guide cover the tool itself, and the Claude Code pricing page covers what the direct plans cost.

LiteLLM also publishes a Claude Code compatibility matrix. A daily automated job runs the Claude Code CLI against the newest LiteLLM release across supported providers, with three model tiers in parallel, and a cell passes only if all three do. That is useful evidence, but it is LiteLLM's own test, not ours.

Codex

Codex reads its configuration from ~/.codex/config.toml. You define LiteLLM as a custom model provider. Codex uses the OpenAI Responses API, which the gateway serves at /v1/responses.

model = "claude-sonnet-5"
model_provider = "litellm"

[model_providers.litellm]
name = "LiteLLM"
base_url = "http://localhost:4000/v1"
env_key = "LITELLM_API_KEY"
wire_api = "responses"

Then export the key and run codex. Notes from the docs:

• env_key names the variable Codex reads the key from, so no secret sits in the file.
• Long agent turns can idle for minutes behind a gateway. The docs suggest raising stream_idle_timeout_ms, stream_max_retries and request_max_retries in the provider block if tasks get cut short.
• Codex only knows metadata for OpenAI's own models. With a gateway name it prints Model metadata for ... not found. Defaulting to fallback metadata. Requests still go through.
• Codex CLI 0.159 or newer can fetch the model list from the gateway, using model_catalog_url on the provider block and the api_key_model_discovery feature flag together. It keeps the catalog for five minutes and ignores catalogs over 1 MiB.
• To keep your ChatGPT sign-in and still route through the gateway, LiteLLM has a separate flow that sends a LiteLLM key alongside your subscription login.

The Codex page in LiteLLM's docs also states that the same config covers Codex inside the ChatGPT desktop app. For the CLI itself, see the Codex CLI listing.

Other agents

LiteLLM's client overview lists Cursor and OpenCode too. The SDK has an agents feature that runs Claude Code, Codex, OpenCode, pi or Deep Agents against a model group on your gateway, with every model call tagged by harness. Our OpenCode and Cursor listings cover those tools, and the OpenRouter listing is the hosted alternative people compare against.

Cost tracking, keys and budgets

LiteLLM tracks spend for all known models automatically, using a model cost map to price each request. Spend is tracked per key, user, team and model, and you can query it.

The basics work like this.

• Virtual keys. You hand these to apps and teammates instead of raw provider keys. Each can carry a budget, rate limits and model access.
• Per-request attribution. Pass a user field to track spend by user. Tags, passed in request metadata, track spend by tag, but the docs mark tag tracking as Enterprise.
• Budgets and limits. A global budget goes in config.yaml under litellm_settings, with max_budget and budget_duration (such as 30d). Teams get budgets and rpm_limit through the /team/new API.
• Reports. The Admin UI shows logs and usage. Programmatic spend reports are an Enterprise item.

Two cautions come straight from the docs. Every budget is enforced against spend read from the database, so none of them cap anything on a database-free deployment. And if the number does not match your provider's bill, LiteLLM has a "debugging a cost discrepancy" guide: align time ranges, compare token categories including cache, then decide whether the gap is ingestion, formula or model-map pricing. You can sync the model pricing data from GitHub to keep it current.

For Codex, the docs say a request has no end user by default, since Codex has no setting that puts one in the body. You add a tracking header to the provider block to attribute requests per developer or project.

Licence: open source core plus Enterprise

The LICENSE file is short. Everything under the enterprise/ directory, if it exists, uses the licence in enterprise/LICENSE. Everything else is MIT, copyright 2023 Berri AI. GitHub's licence badge reads "Other" for that reason, since the repo carries two licences. Our listing records the core as mit.

The Enterprise tier is a licence key, set as LITELLM_LICENSE, on the same gateway image. Nothing to migrate. According to the docs it adds:

Two details are easy to miss. SSO is free for up to 5 users, and beyond that you need an Enterprise licence. And some guardrail integrations need a licence too: llmguard_moderations, llamaguard_moderations, hide_secrets, openai_moderations, google_text_moderation, lakera_prompt_injection and aporia_prompt_injection. Pricing depends on deployment size, and the docs give no number: the Enterprise price is not published. LiteLLM also lists a 30-day trial and hosted options through sales.

Limits and gotchas

• Database or no database. The one-container, no-database path supports only the master key. No virtual keys, no Admin UI model management, no spend tracking, and budgets are not enforced.
• You run it. The gateway is self-hosted, so you own uptime, upgrades and keys. The docs do offer a hosted version through sales.
• Keep the salt key. A changed LITELLM_SALT_KEY makes stored provider keys unreadable until you enter them again. It has no in-place rotation.
• Local only by default. The quickstart script binds to 127.0.0.1. Sharing it with a team means deciding how to expose it and securing that path.
• Model naming. Clients send whatever model id they select. Claude Code's /model picker and Codex's catalog both assume the provider's names unless you configure them.
• Endpoint gaps. The provider table shows endpoint support by provider. Some rows have only chat.
• Two packages. litellm and litellm-core cannot coexist in one environment.
• Vendor claims. Figures such as "8ms P95 latency at 1k RPS" and the adopter logos come from the README and docs index. We did not measure them, and your numbers depend on your hardware and settings.

A short rollout checklist

1. Decide SDK or gateway. If more than one app or person shares models, choose the gateway.
2. Start it with the quickstart script, and back up ~/litellm-gateway/.env.
3. Add one model in the Admin UI and test it in the Playground.
4. Create a virtual key per app or person, each with a budget.
5. Point one client at it, such as Claude Code, and confirm the request in Logs.
6. Add a fallback pair for your most-used model.
7. Check whether you need any Enterprise item before you plan to roll out to more than 5 SSO users.

Our take

LiteLLM earns a place when you have more than one model, more than one person, or a bill you want to understand. The OpenAI-compatible endpoint means very little changes on the client side, and the virtual key idea keeps provider secrets off laptops. For a solo developer who uses one provider, it adds a server for little gain, and the SDK alone may be enough. If you want a hosted router instead, compare our OpenRouter alternatives, and if you are weighing self-hosting in general, see cloud against self-hosted AI.

We have not run LiteLLM ourselves for this page. Every command and setting here comes from the project's README and docs, checked on 31 October 2026, so if a flag has moved by the time you read this, trust the docs. Our Docker MCP Toolkit listing covers a different kind of gateway, one for MCP servers rather than models.