Self-Host an AI Gateway with LiteLLM Proxy: Routing, Budgets and Observability

Every team that ships more than one AI feature eventually hits the same wall. One service calls Anthropic directly, another calls OpenAI, a third uses a local Ollama box. API keys are scattered across four .env files. Nobody can answer "how much did we spend on AI last month, per team?" without exporting three billing CSVs. And when a provider has an outage, every service fails independently because there is no shared fallback.

An AI gateway solves this by putting one OpenAI-compatible endpoint in front of every model you use. Your applications talk to the gateway; the gateway talks to the providers. Routing, retries, budgets, caching and logging all live in one place.

LiteLLM Proxy is the open-source option in this space. Unlike hosted gateways, it runs on your own infrastructure — a container, a Postgres database, and a Redis instance — which matters when you are handling client data under a residency requirement or simply do not want a third party sitting between you and every model call.

This tutorial deploys a production-shaped LiteLLM Proxy and connects a TypeScript application to it.

Prerequisites

Before starting, make sure you have:

• Docker and Docker Compose installed and running
• Node.js 20+ for the TypeScript client
• At least one provider API key (Anthropic, OpenAI, Google, or a local Ollama server)
• Basic familiarity with environment variables and YAML
• A terminal comfortable with curl

No Python knowledge is required. LiteLLM is written in Python, but we run it entirely as a container and interact with it over HTTP.

What You'll Build

A gateway that:

1. Exposes a single OpenAI-compatible endpoint at http://localhost:4000
2. Routes requests across Claude, GPT and a local model behind aliases like smart and cheap
3. Falls back automatically when a provider returns an error or times out
4. Issues virtual keys per team, each with its own monthly budget and rate limit
5. Caches identical requests in Redis to cut spend
6. Logs every call — model, tokens, cost, latency — into Postgres, visible in a built-in admin UI

Then we consume it from TypeScript in two ways: the raw OpenAI SDK, and the Vercel AI SDK for streaming.

Step 1: Project Scaffold

Create a directory for the gateway. It stays separate from your application code — the gateway is infrastructure, not part of any single app.

mkdir ai-gateway && cd ai-gateway
touch docker-compose.yml config.yaml .env

Add the secrets to .env. Never commit this file.

# .env
LITELLM_MASTER_KEY=sk-master-change-me-to-a-long-random-string
LITELLM_SALT_KEY=another-long-random-string-for-encrypting-provider-keys
 
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-proj-...
GEMINI_API_KEY=...
 
POSTGRES_PASSWORD=a-strong-database-password

Two keys deserve explanation. LITELLM_MASTER_KEY is the admin credential — it can create and revoke every other key, so it should never appear in an application. LITELLM_SALT_KEY encrypts provider credentials stored in the database; if you change it later, previously stored provider keys become unreadable, so set it once and back it up.

Step 2: Docker Compose with Postgres and Redis

The proxy alone works without a database, but you lose everything interesting: virtual keys, budgets, and spend logs all require Postgres. Redis powers caching and makes rate limits accurate across multiple proxy replicas.

# docker-compose.yml
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    environment:
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
      DATABASE_URL: postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm
      REDIS_HOST: redis
      REDIS_PORT: "6379"
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      GEMINI_API_KEY: ${GEMINI_API_KEY}
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    restart: unless-stopped
 
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: litellm
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    restart: unless-stopped
 
  redis:
    image: redis:7-alpine
    command: ["redis-server", "--maxmemory", "512mb", "--maxmemory-policy", "allkeys-lru"]
    restart: unless-stopped
 
volumes:
  pgdata:

The Redis eviction policy matters. allkeys-lru means that when the cache fills, the least recently used entries are dropped rather than the server refusing writes — a cache that stops accepting entries silently degrades your gateway into an expensive pass-through.

Step 3: The Config File — Models, Aliases and Fallbacks

config.yaml is where the gateway earns its keep. The core idea: you define public model names that your applications use, and map each one to one or more real provider deployments.

# config.yaml
model_list:
  # A "smart" tier backed by two providers
  - model_name: smart
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 500
  - model_name: smart
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY
      rpm: 500
 
  # A "cheap" tier for classification and summarisation
  - model_name: cheap
    litellm_params:
      model: anthropic/claude-haiku-4-5-20251001
      api_key: os.environ/ANTHROPIC_API_KEY
 
  # A self-hosted model for anything that must not leave the network
  - model_name: private
    litellm_params:
      model: ollama/qwen3:14b
      api_base: http://host.docker.internal:11434
 
  # Embeddings, so vector pipelines route through the gateway too
  - model_name: embed
    litellm_params:
      model: openai/text-embedding-3-large
      api_key: os.environ/OPENAI_API_KEY
 
router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  num_retries: 2
  timeout: 60
  fallbacks:
    - smart: ["cheap"]
  context_window_fallbacks:
    - smart: ["smart"]
 
litellm_settings:
  drop_params: true
  set_verbose: false
 
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Several decisions here are worth unpacking.

Two entries share the name smart. That is deliberate — LiteLLM treats same-named entries as a load-balancing pool. With usage-based-routing-v2, traffic is distributed according to each deployment's remaining rate-limit headroom, tracked in Redis so all replicas agree.

Aliases decouple apps from vendors. Your application asks for smart. Six months from now you swap the underlying model in one YAML file, restart the container, and every service is upgraded. No redeploy, no code change, no coordination meeting.

drop_params: true silently removes parameters a given provider does not support instead of returning a 400. This is what makes "same code, different model" actually work in practice.

Fallbacks fire on errors, not on quality. If every smart deployment fails, the request retries against cheap. That is a degraded answer, but a degraded answer beats a 500 for most user-facing features. Decide per route whether that trade is acceptable.

Start everything:

docker compose up -d
docker compose logs -f litellm

Wait for the log line reporting that the server is running on port 4000, then verify:

curl http://localhost:4000/health/liveliness

Step 4: Your First Request

Test with the master key first, just to confirm routing works end to end:

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{
    "model": "smart",
    "messages": [{"role": "user", "content": "Reply with exactly: gateway ok"}]
  }'

The response is standard OpenAI chat-completion JSON, regardless of which provider actually served it. That compatibility is the whole point: any SDK, tool, or IDE plugin that speaks the OpenAI API now speaks to every model you have configured.

Step 5: Virtual Keys, Teams and Budgets

This is the feature that justifies self-hosting. A virtual key is a credential you mint through the API, scoped to specific models, with a spend cap and rate limits attached.

Create a team first, then a key that belongs to it:

# Create a team with a monthly ceiling
curl http://localhost:4000/team/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_alias": "web-frontend",
    "max_budget": 200,
    "budget_duration": "30d",
    "models": ["smart", "cheap", "embed"]
  }'

The response includes a team_id. Use it to mint a key:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "PASTE_TEAM_ID_HERE",
    "key_alias": "web-frontend-production",
    "max_budget": 50,
    "budget_duration": "30d",
    "rpm_limit": 120,
    "tpm_limit": 200000,
    "metadata": {"service": "noqta-web", "env": "production"}
  }'

You get back a sk-... key. Three limits now apply simultaneously: the key's own 50 USD budget, the team's 200 USD budget, and the per-minute request and token ceilings. When any is exceeded, the gateway returns a 429 with a message naming the limit that tripped — far easier to debug than a generic provider rejection.

A pattern worth adopting: mint one key per service per environment, never one key per developer. When a key leaks, you revoke a single service's credential and rotate it in one place:

curl http://localhost:4000/key/delete \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"keys": ["sk-the-leaked-key"]}'

Check spend at any time:

curl "http://localhost:4000/key/info?key=sk-your-key" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

Step 6: Connecting from TypeScript

Because the gateway is OpenAI-compatible, the official OpenAI SDK works unmodified. Only baseURL and apiKey change.

npm install openai

// lib/gateway.ts
import OpenAI from "openai";
 
export const gateway = new OpenAI({
  baseURL: process.env.LITELLM_BASE_URL ?? "http://localhost:4000/v1",
  apiKey: process.env.LITELLM_API_KEY!, // the virtual key, not the master key
});
 
export async function summarise(text: string) {
  const res = await gateway.chat.completions.create({
    model: "cheap", // alias, not a vendor model id
    messages: [
      { role: "system", content: "Summarise in three bullet points." },
      { role: "user", content: text },
    ],
  });
 
  return res.choices[0]?.message?.content ?? "";
}

Notice that nothing in this file names a vendor. That is the property you are buying: application code that has no opinion about which lab trained the model behind it.

Streaming with the Vercel AI SDK

For Next.js applications, point the AI SDK's OpenAI-compatible provider at the gateway:

npm install ai @ai-sdk/openai-compatible

// app/api/chat/route.ts
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { streamText, convertToModelMessages } from "ai";
 
const gateway = createOpenAICompatible({
  name: "litellm",
  baseURL: process.env.LITELLM_BASE_URL!,
  apiKey: process.env.LITELLM_API_KEY!,
});
 
export async function POST(req: Request) {
  const { messages } = await req.json();
 
  const result = streamText({
    model: gateway.chatModel("smart"),
    messages: convertToModelMessages(messages),
    // Metadata travels to the gateway and lands in the spend logs
    headers: {
      "x-litellm-tags": "feature:support-chat,tier:pro",
    },
  });
 
  return result.toUIMessageStreamResponse();
}

Those tags are more useful than they look. Attribution stops being "the web app spent 340 USD" and becomes "support chat spent 210 USD, onboarding spent 90 USD" — which is the granularity you need to decide what to optimise.

Step 7: Caching

Identical prompts are more common than teams expect: retries, refreshed pages, batch jobs re-run over unchanged rows. Redis caching turns those into free, instant responses.

Add to config.yaml:

litellm_settings:
  drop_params: true
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    ttl: 3600
    supported_call_types: ["acompletion", "atext_completion", "aembedding"]

Restart with docker compose restart litellm. Send the same request twice and compare — the second response returns in milliseconds and records zero cost.

Cache per request when a given call must always be fresh:

const res = await gateway.chat.completions.create(
  { model: "cheap", messages },
  { headers: { "x-litellm-no-cache": "true" } },
);

Embeddings are where caching pays off most. Re-indexing a document set usually means re-embedding text that has not changed; with a cache in front, only genuinely new chunks reach the provider.

Step 8: Observability

Every request already lands in Postgres. The built-in UI at http://localhost:4000/ui — log in with the master key — shows spend per key, per team, per model, plus error rates and latency percentiles.

For deeper tracing, add callbacks that forward request metadata to an observability platform:

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

Supply LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY as environment variables in the compose file. Every call through the gateway then produces a trace with prompt, completion, token counts, cost and latency — without a single line of instrumentation in your application. If you already run Langfuse, our Langfuse LLM observability tutorial covers the dashboard side.

For alerting, the gateway can post budget and error notifications to Slack:

general_settings:
  alerting: ["slack"]
  alerting_threshold: 300  # seconds; flags requests that hang

Testing Your Implementation

Work through these checks before pointing production traffic at the gateway.

Routing. Call smart twenty times and confirm in the UI that requests are spread across both deployments rather than pinned to one.

Fallback. Temporarily set an invalid api_key on the Anthropic entry, restart, and send a request. It should succeed via OpenAI, and the logs should show a retry.

Budget enforcement. Mint a key with "max_budget": 0.01, send requests until it trips, and confirm you receive a 429 naming the budget.

Rate limits. Mint a key with "rpm_limit": 2 and fire five requests in a loop:

for i in $(seq 1 5); do
  curl -s -o /dev/null -w "%{http_code}\n" http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $TEST_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"cheap","messages":[{"role":"user","content":"hi"}]}'
done

You should see two 200 responses followed by 429s.

Cache. Send an identical request twice and compare latency and recorded cost.

Troubleshooting

The container starts, then exits. Almost always the database. Check docker compose logs postgres and confirm DATABASE_URL matches the Postgres credentials exactly — a mismatched password produces a confusing startup error rather than an obvious auth failure.

Invalid model name passed in. The model name in your request must match a model_name in config.yaml, not the provider's model id. Requesting claude-opus-4-8 fails if you defined the alias smart. List what is available with curl http://localhost:4000/v1/models.

Ollama is unreachable from the container. Inside Docker, localhost is the container itself. Use http://host.docker.internal:11434 on macOS and Windows; on Linux add extra_hosts: ["host.docker.internal:host-gateway"] to the service.

Budgets never reset. budget_duration is a rolling window from key creation, not a calendar month. A key created on the 20th resets on the 20th.

Spend shows as zero for a custom model. LiteLLM prices requests from a built-in cost map. Self-hosted or unusual models need explicit pricing:

  - model_name: private
    litellm_params:
      model: ollama/qwen3:14b
      api_base: http://host.docker.internal:11434
    model_info:
      input_cost_per_token: 0.0000001
      output_cost_per_token: 0.0000002

Streaming stalls behind a reverse proxy. Nginx buffers responses by default, which breaks server-sent events. Set proxy_buffering off; on the gateway location block.

Going to Production

A few changes separate the local setup above from something you would trust with real traffic:

1. Terminate TLS. Put the proxy behind Caddy, Nginx or a load balancer. Never expose port 4000 directly.
2. Run more than one replica. The proxy is stateless; Redis keeps rate limits and cache consistent across instances.
3. Use managed Postgres. The spend ledger is your billing record — it deserves real backups.
4. Pin the image tag. main-stable moves. Pin a specific release and upgrade deliberately.
5. Rotate the master key on a schedule and keep it out of every application environment.
6. Set alerts on the budget, not just on errors. Cost regressions are usually silent.

Next Steps

• Route your existing agents through the gateway — the Vercel AI SDK and Mastra tutorials both use configurable base URLs
• Add Langfuse for prompt versioning alongside gateway-level tracing
• Compare with the managed path in our Vercel AI Gateway tutorial
• Enable guardrail plugins for PII redaction before requests reach external providers

Conclusion

The gateway pattern is unglamorous infrastructure that quietly removes a whole category of problems. Provider keys stop spreading through your codebase. Model choice becomes a configuration change rather than a deployment. Spend becomes attributable to the feature that caused it. Outages degrade instead of failing.

Self-hosting LiteLLM adds an hour of setup and a container to maintain, and in exchange your prompts and completions never traverse infrastructure you do not control — which, for teams handling client data under residency or confidentiality obligations, is not a preference but a requirement.

Start with one service. Point it at the gateway, watch the logs for a week, then migrate the next one.