Cloudflare Workers AI: Run LLMs at the Edge with TypeScript

Prerequisites

Before starting, make sure you have:

• A Cloudflare account (free tier works for this tutorial)
• Node.js 20+ installed
• Wrangler CLI: npm install -g wrangler
• Basic TypeScript and async/await knowledge
• Familiarity with REST APIs

What You'll Build

By the end of this tutorial, you'll have a fully functional edge AI API that:

• Generates text using Meta Llama 3 running on Cloudflare's GPU network
• Streams responses in real time via Server-Sent Events
• Creates semantic text embeddings using BAAI/bge
• Stores and queries vectors in Cloudflare Vectorize
• Implements a complete RAG (Retrieval-Augmented Generation) pipeline

All of this runs on Cloudflare's global network — no GPU servers to manage, no cold starts, and inference happening within milliseconds of your users.

Why Cloudflare Workers AI?

Running AI inference at the edge solves several problems that cloud-hosted model APIs struggle with.

Zero infrastructure management — Workers AI runs on Cloudflare's serverless GPU network. You write TypeScript, deploy with one command, and Cloudflare handles model loading, GPU allocation, and scaling.

Global low-latency — Inference runs at the edge location nearest your user. For MENA users, this means Cloudflare's data centers in the UAE, Saudi Arabia, and Europe handle requests directly rather than routing to US-based API endpoints.

Predictable pricing — Workers AI charges per neuron (compute unit), not per token like most LLM APIs. This makes cost modeling straightforward for high-throughput applications.

No API key juggling — Models are available via your Workers AI binding (env.AI). No external API keys required for inference.

Step 1: Create a Workers Project

Use the official Cloudflare scaffold to create a new project:

npm create cloudflare@latest ai-edge-api

When prompted:

• What type of application? Select "Hello World" Worker
• Do you want to use TypeScript? Yes
• Do you want to deploy? No (we'll deploy in Step 10)

Navigate to the project:

cd ai-edge-api

The scaffold creates these key files:

ai-edge-api/
├── src/
│   └── index.ts        # Your Worker entry point
├── wrangler.toml       # Cloudflare configuration
├── package.json
└── tsconfig.json

Step 2: Configure the Workers AI Binding

Open wrangler.toml and add the AI binding. This is all that's needed — no API keys:

name = "ai-edge-api"
main = "src/index.ts"
compatibility_date = "2024-01-01"
 
[ai]
binding = "AI"

Install the Workers types package for TypeScript autocompletion:

npm install --save-dev @cloudflare/workers-types

Update tsconfig.json to include Workers types:

{
  "compilerOptions": {
    "target": "ES2022",
    "lib": ["ES2022"],
    "module": "ES2022",
    "moduleResolution": "bundler",
    "types": ["@cloudflare/workers-types"],
    "strict": true
  }
}

Step 3: First Inference — Text Generation

Replace the contents of src/index.ts with this basic text generation Worker:

export interface Env {
  AI: Ai;
}
 
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    const prompt = url.searchParams.get("prompt") ?? "Tell me about edge computing";
 
    const messages = [
      {
        role: "system",
        content: "You are a helpful assistant specializing in technology.",
      },
      {
        role: "user",
        content: prompt,
      },
    ];
 
    const response = await env.AI.run(
      "@cf/meta/llama-3-8b-instruct",
      { messages }
    );
 
    return Response.json(response);
  },
} satisfies ExportedHandler<Env>;

Test it locally with Wrangler:

npx wrangler dev --remote

The --remote flag is required for Workers AI — the local simulator cannot load GPU models, so dev mode connects to Cloudflare's network. Open http://localhost:8787?prompt=What+is+edge+computing and you should see a JSON response from Llama 3:

{
  "response": "Edge computing is a distributed computing paradigm that brings computation closer to data sources..."
}

Step 4: Streaming Responses

For long-form generation, streaming is essential — users see text appear progressively rather than waiting for the complete response. Workers AI supports streaming via Server-Sent Events (SSE):

export interface Env {
  AI: Ai;
}
 
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    const path = url.pathname;
 
    if (path === "/stream") {
      const prompt =
        url.searchParams.get("prompt") ?? "Explain the benefits of edge AI";
 
      const messages = [
        {
          role: "system",
          content: "You are a concise technical writer. Respond in 3-4 sentences.",
        },
        { role: "user", content: prompt },
      ];
 
      const stream = await env.AI.run(
        "@cf/meta/llama-3-8b-instruct",
        { messages, stream: true }
      );
 
      return new Response(stream, {
        headers: {
          "content-type": "text/event-stream",
          "cache-control": "no-cache",
          "access-control-allow-origin": "*",
        },
      });
    }
 
    return new Response("Workers AI API — try /stream?prompt=your+question");
  },
} satisfies ExportedHandler<Env>;

Test the streaming endpoint:

curl "http://localhost:8787/stream?prompt=What+is+RAG?"

You'll see SSE events flowing in real time:

data: {"response":"RAG"}
data: {"response":" stands"}
data: {"response":" for"}
data: {"response":" Retrieval-Augmented"}
...
data: [DONE]

Step 5: Text Embeddings

Embeddings convert text into high-dimensional vectors that capture semantic meaning. Similar content produces similar vectors — the foundation of semantic search and RAG. Workers AI includes @cf/baai/bge-base-en-v1.5, a 768-dimension embedding model optimized for English text.

Add an embedding endpoint to your Worker:

if (path === "/embed") {
  if (request.method !== "POST") {
    return new Response("POST required", { status: 405 });
  }
 
  const body = await request.json<{ texts: string[] }>();
  const { texts } = body;
 
  if (!texts || texts.length === 0) {
    return Response.json({ error: "texts array required" }, { status: 400 });
  }
 
  const embeddings = await env.AI.run(
    "@cf/baai/bge-base-en-v1.5",
    { text: texts }
  );
 
  return Response.json({
    model: "@cf/baai/bge-base-en-v1.5",
    dimensions: embeddings.shape[1],
    count: embeddings.data.length,
    embeddings: embeddings.data,
  });
}

Test with curl:

curl -X POST "http://localhost:8787/embed" \
  -H "Content-Type: application/json" \
  -d '{"texts": ["Cloudflare Workers runs TypeScript at the edge", "Edge computing reduces latency for global users"]}'

The response includes 768-dimension float arrays for each input text. Passing an array of texts in a single call is more efficient than calling the model in a loop.

Step 6: Set Up Vectorize for Semantic Search

Vectorize is Cloudflare's managed vector database — it stores embeddings and finds nearest neighbors in milliseconds. Create an index via Wrangler:

npx wrangler vectorize create notes-index --dimensions=768 --metric=cosine

The --dimensions=768 matches the BAAI embedding model output. The cosine metric finds vectors that point in the same semantic direction, regardless of magnitude.

Add the Vectorize binding to wrangler.toml:

[ai]
binding = "AI"
 
[[vectorize]]
binding = "VECTORIZE"
index_name = "notes-index"

Update your Env interface:

export interface Env {
  AI: Ai;
  VECTORIZE: VectorizeIndex;
}

Step 7: Build a RAG Pipeline

RAG (Retrieval-Augmented Generation) combines vector search with LLM generation. The pattern has three stages:

1. Ingest — embed documents and store vectors in Vectorize
2. Retrieve — embed the user question and find similar document vectors
3. Generate — pass retrieved documents as context to the LLM

Here is a complete implementation:

export interface Env {
  AI: Ai;
  VECTORIZE: VectorizeIndex;
}
 
interface TextGenResponse {
  response: string;
}
 
interface EmbeddingResponse {
  shape: number[];
  data: number[][];
}
 
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    const path = url.pathname;
 
    // POST /notes — ingest a document
    if (path === "/notes" && request.method === "POST") {
      const { text } = await request.json<{ text: string }>();
      const id = crypto.randomUUID();
 
      // Generate embedding
      const embeddings = (await env.AI.run(
        "@cf/baai/bge-base-en-v1.5",
        { text: [text] }
      )) as EmbeddingResponse;
 
      // Upsert vector into Vectorize with text as metadata
      await env.VECTORIZE.upsert([
        {
          id,
          values: embeddings.data[0],
          metadata: { text },
        },
      ]);
 
      return Response.json({ id, message: "Note ingested successfully" });
    }
 
    // GET /ask?q=... — RAG query
    if (path === "/ask") {
      const question = url.searchParams.get("q");
      if (!question) {
        return Response.json({ error: "q parameter required" }, { status: 400 });
      }
 
      // Step 1: Embed the question
      const questionEmbedding = (await env.AI.run(
        "@cf/baai/bge-base-en-v1.5",
        { text: [question] }
      )) as EmbeddingResponse;
 
      // Step 2: Find relevant notes (top 3, score above 0.6)
      const matches = await env.VECTORIZE.query(
        questionEmbedding.data[0],
        { topK: 3, returnMetadata: "all" }
      );
 
      const contextChunks = matches.matches
        .filter((m) => m.score > 0.6)
        .map((m) => m.metadata?.text as string)
        .filter(Boolean);
 
      const context = contextChunks.join("\n\n");
 
      // Step 3: Build system prompt with retrieved context
      const systemPrompt =
        context.length > 0
          ? `You are a helpful assistant. Use the following context to answer accurately. If the context does not contain the answer, say so.\n\nContext:\n${context}`
          : "You are a helpful assistant. Answer based on your general knowledge.";
 
      // Step 4: Generate answer
      const aiResponse = (await env.AI.run(
        "@cf/meta/llama-3-8b-instruct",
        {
          messages: [
            { role: "system", content: systemPrompt },
            { role: "user", content: question },
          ],
        }
      )) as TextGenResponse;
 
      return Response.json({
        question,
        answer: aiResponse.response,
        sources: contextChunks.length,
        topScore: matches.matches[0]?.score ?? 0,
      });
    }
 
    return Response.json({
      endpoints: {
        "POST /notes": "Ingest a document",
        "GET /ask?q=": "Ask a question via RAG",
        "GET /stream?prompt=": "Stream a response",
        "POST /embed": "Generate embeddings",
      },
    });
  },
} satisfies ExportedHandler<Env>;

Test the full RAG pipeline:

# Ingest knowledge
curl -X POST "http://localhost:8787/notes" \
  -H "Content-Type: application/json" \
  -d '{"text": "Cloudflare Workers AI runs Meta Llama 3, Mistral, and Gemma models directly on Cloudflare edge nodes globally."}'
 
curl -X POST "http://localhost:8787/notes" \
  -H "Content-Type: application/json" \
  -d '{"text": "Vectorize is Cloudflare'\''s managed vector database with cosine, dot product, and Euclidean distance metrics."}'
 
# Ask a question — the LLM answers using your ingested notes
curl "http://localhost:8787/ask?q=What+models+does+Workers+AI+support?"

Step 8: Available Models

Workers AI includes over 50 models across several categories:

Text Generation:

• @cf/meta/llama-3-8b-instruct — Fast general-purpose chat model
• @cf/meta/llama-3.2-11b-vision-instruct — Multimodal with image understanding
• @cf/mistral/mistral-7b-instruct-v0.1 — Strong at instruction following
• @hf/google/gemma-7b-it — Google instruction-tuned model

Embeddings:

• @cf/baai/bge-base-en-v1.5 — 768 dimensions, English optimized
• @cf/baai/bge-large-en-v1.5 — 1024 dimensions, higher accuracy

Image Generation:

• @cf/stabilityai/stable-diffusion-xl-base-1.0 — SDXL for high-quality images
• @cf/lykon/dreamshaper-8-lcm — Fast generation with LCM sampler

Speech Synthesis:

• @cf/myshell-ai/melotts — Text-to-speech, multiple languages

You can browse the full catalog at developers.cloudflare.com/workers-ai/models/.

Step 9: Type-Safe Model Responses

Different models return different response shapes. Always cast your results for type safety:

interface TextGenResponse {
  response: string;
  p?: string;
}
 
interface EmbeddingResponse {
  shape: number[];   // e.g. [2, 768]
  data: number[][];  // one float array per input text
}
 
// Text generation
const textResult = (await env.AI.run(
  "@cf/meta/llama-3-8b-instruct",
  { messages }
)) as TextGenResponse;
 
console.log(textResult.response); // "Hello! How can I help..."
 
// Embeddings
const embResult = (await env.AI.run(
  "@cf/baai/bge-base-en-v1.5",
  { text: ["hello world"] }
)) as EmbeddingResponse;
 
console.log(embResult.shape);         // [1, 768]
console.log(embResult.data[0].length); // 768

Step 10: Deploy to Cloudflare

Authenticate Wrangler with your Cloudflare account:

npx wrangler login

Then deploy with a single command:

npx wrangler deploy

Wrangler uploads your Worker and deploys it globally. Your endpoint will be:

https://ai-edge-api.YOUR-SUBDOMAIN.workers.dev

For a custom domain, add a route in wrangler.toml:

[[routes]]
pattern = "api.yourdomain.com/ai/*"
zone_name = "yourdomain.com"

Testing Your Deployment

Once live, run a quick end-to-end test:

BASE="https://ai-edge-api.YOUR-SUBDOMAIN.workers.dev"
 
# Ingest a note
curl -X POST "$BASE/notes" \
  -H "Content-Type: application/json" \
  -d '{"text": "Tunisia is a North African country with a rapidly growing tech sector."}'
 
# Stream a response
curl "$BASE/stream?prompt=What+makes+edge+AI+different+from+cloud+AI?"
 
# Ask via RAG
curl "$BASE/ask?q=Tell+me+about+Tunisia"

Troubleshooting

"model not found" error — Check the exact model ID against the Workers AI catalog. Model IDs follow the pattern @org/model-name and are case-sensitive. A typo like @cf/meta/llama-3-8b instead of @cf/meta/llama-3-8b-instruct will fail.

Vectorize binding missing — Run npx wrangler vectorize list to confirm your index exists. The binding name in wrangler.toml must match the field name in your Env interface exactly.

Local dev returns 503 — Workers AI requires --remote in local development. The local simulator cannot load GPU models. Always use npx wrangler dev --remote for AI features.

Embedding dimension mismatch — Always match your embedding model's output dimensions to the Vectorize index. bge-base-en-v1.5 produces 768 dimensions; bge-large-en-v1.5 produces 1024. These indexes are not interchangeable.

Low RAG accuracy — The 0.6 cosine similarity threshold is a starting point. For domain-specific content, you may need to lower it to 0.5 for broader recall, or raise it to 0.75 for higher precision. Monitor topScore in your responses to calibrate.

Performance Considerations

• Batch embeddings — Pass an array of texts to env.AI.run rather than calling it in a loop. A single batched call is significantly faster than sequential calls.
• Cache static embeddings — Document embeddings don't change unless the document changes. Generate them once at ingest time, not at query time.
• Vectorize topK — Request only as many results as you'll use in context. Values between 3 and 5 are usually sufficient for RAG without inflating the LLM prompt.
• Choose the right model — llama-3-8b-instruct is faster and cheaper than llama-3.2-11b-vision-instruct. Use the lighter model unless vision capabilities are required.

Next Steps

Now that you have a working edge AI stack, consider these extensions:

• Add D1 persistence — Replace ephemeral state with a Cloudflare D1 SQLite database for durable note storage
• Add routing with Hono — Use the Hono framework for cleaner routing, middleware, and Zod validation instead of manual URL parsing
• Multi-modal RAG — Use llama-3.2-11b-vision-instruct to add image understanding to your pipeline
• Semantic caching — Cache LLM responses for semantically similar questions using Vectorize to cut costs on repeated queries
• Rate limiting with Workers KV — Protect your AI endpoint by storing per-user request counts in Cloudflare KV

Conclusion

Cloudflare Workers AI gives you a production-grade edge AI stack with zero infrastructure overhead. You've built a complete pipeline: text generation, real-time streaming, semantic embeddings, Vectorize vector storage, and a full RAG system — all in TypeScript, all running at the edge.

The embed-store-retrieve-generate pattern you've implemented here is the foundation of virtually every AI-powered knowledge base, documentation search, and customer support system. The same approach applies to product catalogs, internal wikis, or any corpus of text your application needs to reason over.