Noqta
writing/tutorial/2026/07
● TutorialJul 27, 2026·30 min read

Building an Arabic RAG Pipeline with Text Normalization in TypeScript

Learn how to build a production-ready Arabic RAG pipeline in TypeScript. Covers Unicode normalization, multilingual embeddings, LanceDB vector search, and Arabic answer generation with Claude Sonnet 5.

AI Bot
AI Bot
Author
·EN · FR · AR

Prerequisites

Before starting, you need:

  • Node.js 20 or later
  • TypeScript 5+
  • OpenAI API key (for embeddings)
  • Anthropic API key (for generation with Claude Sonnet 5)
  • Basic familiarity with Next.js 15 App Router

What You'll Build

By the end of this tutorial you will have a full Arabic RAG pipeline that:

  1. Normalizes Arabic text — removing diacritics, resolving Alef variants, and stripping stylistic characters
  2. Generates multilingual embeddings using OpenAI text-embedding-3-small
  3. Stores and retrieves document chunks using LanceDB
  4. Generates fluent Arabic answers using Claude Sonnet 5

The pipeline exposes a single Next.js API route at /api/arabic-rag that accepts a question in Arabic (or any language) and returns a grounded answer with source citations.

Why Arabic Needs Normalization

Arabic is written in a rich Unicode subset that allows many representations of the same word. Without normalization, retrieval fails silently — the query and the indexed document match semantically but differ byte-by-byte, producing zero recall from an otherwise correct vector search.

The four biggest sources of mismatch in Arabic text:

1. Alef variants. The letter Alef appears in four Unicode code points: plain Alef (ا), Alef with Hamza above (أ), Alef with Hamza below (إ), and Alef with Madda (آ). A search for "إسلام" misses "اسلام" if these variants are not normalized to the same form.

2. Tashkeel (diacritics). Formal texts include vowel marks — fatha, damma, kasra, sukun, shadda, and tanwin forms. Informal texts omit them entirely. "كِتَابٌ" and "كتاب" are the same word but produce different embedding vectors without normalization.

3. Tatweel. The character ـ is a kashida used to stretch letters for aesthetic purposes (like "كتاب" written as "كتاااب"). It carries no semantic meaning and must be stripped before embedding.

4. Alef Maqsurah. The letter (ى) is often confused with (ي) in informal writing. Without normalization, "على" and "علي" hash to different vectors even though in many contexts they represent the same word.

Step 1: Project Setup

Create a Next.js 15 app with TypeScript:

npx create-next-app@latest arabic-rag --typescript --app --src-dir --turbopack
cd arabic-rag

Install the required packages:

npm install @lancedb/lancedb openai @anthropic-ai/sdk
npm install -D tsx

Add API keys to .env.local:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

Step 2: The Arabic Normalizer

Create src/lib/arabic-normalizer.ts. This module is the foundation of the pipeline — every document and every query passes through it before being embedded.

const TASHKEEL = /[ؐ-ًؚ-ٰٟ]/g;
const TATWEEL = /ـ/g;
const ALEF_VARIANTS = /[آأإٱ]/g;
const HAMZA_ON_WAW = /ؤ/g;
const HAMZA_ON_YA = /ئ/g;
const TEH_MARBUTA = /ة/g;
const ALEF_MAQSURAH = /ى/g;
const EXTRA_SPACES = /\s+/g;
 
export interface NormalizerOptions {
  removeTashkeel?: boolean;
  removeTatweel?: boolean;
  normalizeAlef?: boolean;
  normalizeHamza?: boolean;
  normalizeTehMarbuta?: boolean;
  normalizeAlefMaqsurah?: boolean;
}
 
const DEFAULT_OPTIONS: NormalizerOptions = {
  removeTashkeel: true,
  removeTatweel: true,
  normalizeAlef: true,
  normalizeHamza: false,
  normalizeTehMarbuta: false,
  normalizeAlefMaqsurah: true,
};
 
export function normalizeArabic(
  text: string,
  options: NormalizerOptions = DEFAULT_OPTIONS
): string {
  let result = text.normalize("NFKC");
 
  if (options.removeTashkeel) result = result.replace(TASHKEEL, "");
  if (options.removeTatweel) result = result.replace(TATWEEL, "");
  if (options.normalizeAlef) result = result.replace(ALEF_VARIANTS, "ا");
  if (options.normalizeHamza) {
    result = result.replace(HAMZA_ON_WAW, "و");
    result = result.replace(HAMZA_ON_YA, "ي");
  }
  if (options.normalizeTehMarbuta)
    result = result.replace(TEH_MARBUTA, "ه");
  if (options.normalizeAlefMaqsurah)
    result = result.replace(ALEF_MAQSURAH, "ي");
 
  return result.replace(EXTRA_SPACES, " ").trim();
}
 
export const normalizeForIndexing = (text: string) =>
  normalizeArabic(text, DEFAULT_OPTIONS);
 
export const normalizeForQuery = (text: string) =>
  normalizeArabic(text, DEFAULT_OPTIONS);

Hamza normalization is disabled by default. Normalizing (ؤ) to (و) and (ئ) to (ي) can conflate semantically distinct words. Enable normalizeHamza: true only for dialectal Arabic corpora where Hamza is consistently omitted in the source data.

Teh Marbuta normalization is disabled by default. Mapping (ة) to (ه) improves keyword recall but can merge unrelated words. Multilingual embedding models handle Teh Marbuta at the semantic level without requiring this normalization.

Step 3: Embedding Pipeline

Create src/lib/embeddings.ts:

import OpenAI from "openai";
 
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
 
export const EMBEDDING_MODEL = "text-embedding-3-small";
export const EMBEDDING_DIMENSIONS = 1536;
 
export async function embedText(text: string): Promise<number[]> {
  const res = await openai.embeddings.create({
    model: EMBEDDING_MODEL,
    input: text,
    encoding_format: "float",
  });
  return res.data[0].embedding;
}
 
export async function embedBatch(texts: string[]): Promise<number[][]> {
  if (texts.length === 0) return [];
  const res = await openai.embeddings.create({
    model: EMBEDDING_MODEL,
    input: texts,
    encoding_format: "float",
  });
  return res.data.map((d) => d.embedding);
}

text-embedding-3-small was trained on multilingual data and understands Arabic semantics without normalization. Normalization still matters because it ensures "كِتَاب" and "كتاب" land in the same vector neighborhood rather than producing measurable cosine distance between two representations of the same word. In practice, normalization improves Arabic RAG recall by 15-30% on typical corpora.

Step 4: Vector Store

Create src/lib/vector-store.ts. LanceDB uses schema inference — passing a typed array of documents on first write is enough to define the table schema. No Apache Arrow imports needed.

import * as lancedb from "@lancedb/lancedb";
 
export interface ArabicDocument {
  id: string;
  text: string;
  normalizedText: string;
  vector: number[];
  source: string;
  metadata: string;
}
 
let conn: lancedb.Connection | null = null;
 
async function getConn(): Promise<lancedb.Connection> {
  if (!conn) conn = await lancedb.connect(".lancedb");
  return conn;
}
 
export async function addDocuments(
  docs: ArabicDocument[],
  tableName = "arabic_docs"
): Promise<void> {
  const db = await getConn();
  const tables = await db.tableNames();
 
  if (tables.includes(tableName)) {
    const table = await db.openTable(tableName);
    await table.add(docs);
  } else {
    await db.createTable(tableName, docs);
  }
}
 
export async function searchDocuments(
  queryVector: number[],
  limit = 5,
  tableName = "arabic_docs"
): Promise<ArabicDocument[]> {
  const db = await getConn();
  const table = await db.openTable(tableName);
  return (await table.search(queryVector).limit(limit).toArray()) as ArabicDocument[];
}

The .lancedb directory is a folder on disk — no daemon, no network call, no setup beyond npm install. For production deployments, replace the path string with an S3 URI and pass storageOptions with your AWS credentials.

LanceDB infers the vector column from the vector field automatically. Using the conventional name vector avoids any need to specify metricType or column names on the search call.

Step 5: Indexing Pipeline

Create src/lib/indexer.ts:

import { normalizeForIndexing } from "./arabic-normalizer";
import { embedBatch } from "./embeddings";
import { addDocuments, type ArabicDocument } from "./vector-store";
 
export interface RawDocument {
  text: string;
  source: string;
  metadata?: Record<string, string>;
}
 
function chunkArabicText(text: string, maxChars = 800): string[] {
  // Split on sentence-ending punctuation common in Arabic text.
  // Note: Arabic question mark ؟ (U+061F) must be included alongside ?
  const sentences = text
    .split(/[.!?؟\n]+/)
    .map((s) => s.trim())
    .filter((s) => s.length > 20);
 
  const chunks: string[] = [];
  let current = "";
 
  for (const sentence of sentences) {
    const candidate = current ? current + " " + sentence : sentence;
    if (candidate.length > maxChars && current) {
      chunks.push(current);
      current = sentence;
    } else {
      current = candidate;
    }
  }
  if (current) chunks.push(current);
  return chunks;
}
 
export async function indexDocuments(
  rawDocs: RawDocument[],
  batchSize = 20
): Promise<void> {
  type Chunk = {
    text: string;
    normalized: string;
    source: string;
    meta: string;
  };
 
  const allChunks: Chunk[] = [];
 
  for (const doc of rawDocs) {
    const chunks = chunkArabicText(doc.text);
    for (const chunk of chunks) {
      allChunks.push({
        text: chunk,
        normalized: normalizeForIndexing(chunk),
        source: doc.source,
        meta: JSON.stringify(doc.metadata ?? {}),
      });
    }
  }
 
  for (let i = 0; i < allChunks.length; i += batchSize) {
    const batch = allChunks.slice(i, i + batchSize);
    const embeddings = await embedBatch(batch.map((c) => c.normalized));
 
    const docs: ArabicDocument[] = batch.map((chunk, j) => ({
      id: crypto.randomUUID(),
      text: chunk.text,
      normalizedText: chunk.normalized,
      vector: embeddings[j],
      source: chunk.source,
      metadata: chunk.meta,
    }));
 
    await addDocuments(docs);
    process.stderr.write(
      `Indexed ${i + batch.length}/${allChunks.length} chunks\n`
    );
  }
}

A chunk target of 800 characters covers roughly 150-200 Arabic words — similar semantic density to a 300-word English chunk at 1,100 characters. The Arabic question mark (U+061F) must be in the sentence splitter because Arabic authors mix Latin and Arabic punctuation in the same text.

Step 6: Retrieval

Create src/lib/retriever.ts:

import { normalizeForQuery } from "./arabic-normalizer";
import { embedText } from "./embeddings";
import { searchDocuments } from "./vector-store";
 
export interface RetrievalResult {
  text: string;
  source: string;
  rank: number;
}
 
export async function retrieve(
  query: string,
  topK = 5
): Promise<RetrievalResult[]> {
  const normalizedQuery = normalizeForQuery(query);
  const queryVector = await embedText(normalizedQuery);
  const results = await searchDocuments(queryVector, topK);
 
  return results.map((doc, i) => ({
    text: doc.text,
    source: doc.source,
    rank: i + 1,
  }));
}

The retriever normalizes the query with the same function used at index time. This is the single most important invariant in the pipeline. If query normalization diverges from index normalization — even by one option flag — recall degrades silently. Queries return results, but the right documents are missing with no error to diagnose.

Step 7: Generation with Claude Sonnet 5

Create src/lib/generator.ts:

import Anthropic from "@anthropic-ai/sdk";
import type { RetrievalResult } from "./retriever";
 
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
 
type Lang = "ar" | "en" | "fr";
 
const SYSTEM_PROMPTS: Record<Lang, string> = {
  ar: "أنت مساعد متخصص يجيب على الأسئلة بالعربية الفصحى الواضحة. استند فقط إلى السياق المقدم. إذا لم يتضمن السياق الإجابة، فأخبر المستخدم بذلك صراحةً.",
  en: "You are a specialized assistant that answers questions in clear English. Base your answers only on the provided context. If the context does not contain the answer, say so explicitly.",
  fr: "Tu es un assistant spécialisé qui répond en français clair. Base-toi uniquement sur le contexte fourni. Si le contexte ne contient pas la réponse, dis-le explicitement.",
};
 
export async function generateAnswer(
  query: string,
  context: RetrievalResult[],
  lang: Lang = "ar"
): Promise<string> {
  const contextText = context
    .map((r) => `[${r.rank}] ${r.text}\nالمصدر: ${r.source}`)
    .join("\n\n---\n\n");
 
  const userPrompt =
    lang === "ar"
      ? `السياق:\n\n${contextText}\n\nالسؤال: ${query}`
      : `Context:\n\n${contextText}\n\nQuestion: ${query}`;
 
  const response = await anthropic.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 1024,
    system: SYSTEM_PROMPTS[lang],
    messages: [{ role: "user", content: userPrompt }],
  });
 
  const block = response.content[0];
  if (block.type !== "text") throw new Error("Unexpected content block type");
  return block.text;
}

Step 8: Next.js API Route

Create src/app/api/arabic-rag/route.ts:

import { NextRequest, NextResponse } from "next/server";
import { retrieve } from "@/lib/retriever";
import { generateAnswer } from "@/lib/generator";
 
export const runtime = "nodejs";
export const maxDuration = 30;
 
export async function POST(req: NextRequest) {
  try {
    const { query, lang = "ar", topK = 5 } = (await req.json()) as {
      query: string;
      lang?: "ar" | "en" | "fr";
      topK?: number;
    };
 
    if (!query || typeof query !== "string") {
      return NextResponse.json({ error: "query is required" }, { status: 400 });
    }
 
    const context = await retrieve(query, topK);
 
    if (context.length === 0) {
      return NextResponse.json(
        {
          answer:
            lang === "ar"
              ? "لم أجد معلومات كافية للإجابة على سؤالك."
              : "No relevant documents found.",
          context: [],
        },
        { status: 200 }
      );
    }
 
    const answer = await generateAnswer(query, context, lang);
    return NextResponse.json({ answer, context });
  } catch (err) {
    console.error("[arabic-rag]", err);
    return NextResponse.json(
      { error: "Internal server error" },
      { status: 500 }
    );
  }
}

runtime = "nodejs" is mandatory — LanceDB uses native Rust bindings that cannot run on the Vercel Edge runtime or in a Cloudflare Worker.

Step 9: Seeding Test Data

Create scripts/seed.ts:

import { indexDocuments } from "../src/lib/indexer";
 
const sampleDocs = [
  {
    text: "الذكاء الاصطناعي هو محاكاة للذكاء البشري في الآلات التي تم برمجتها للتفكير مثل البشر وتقليد أفعالهم. يشمل هذا المصطلح أي آلة تُظهر سمات مرتبطة بالعقل البشري مثل التعلم وحل المشكلات.",
    source: "مقدمة إلى الذكاء الاصطناعي",
    metadata: { category: "تقنية" },
  },
  {
    text: "التعلم الآلي هو تطبيق للذكاء الاصطناعي يوفر للأنظمة القدرة على التعلم التلقائي والتحسن من الخبرة دون أن تتم برمجتها صراحةً. يركز على تطوير برامج الكمبيوتر التي يمكنها الوصول إلى البيانات واستخدامها للتعلم الذاتي.",
    source: "أساسيات التعلم الآلي",
    metadata: { category: "تقنية" },
  },
  {
    text: "معالجة اللغات الطبيعية هي فرع من فروع الذكاء الاصطناعي يتعلق بالتفاعل بين أجهزة الكمبيوتر والبشر باستخدام اللغة الطبيعية. الهدف النهائي هو قراءة النصوص وفهمها والاستجابة لها بطريقة ذات معنى.",
    source: "معالجة اللغات الطبيعية",
    metadata: { category: "تقنية" },
  },
];
 
indexDocuments(sampleDocs)
  .then(() => {
    process.stderr.write("Seeding complete\n");
    process.exit(0);
  })
  .catch((err) => {
    console.error(err);
    process.exit(1);
  });

Run the seeder then test:

npx tsx scripts/seed.ts
npm run dev
curl -X POST http://localhost:3000/api/arabic-rag \
  -H "Content-Type: application/json" \
  -d '{"query":"ما هو التعلم الآلي؟","lang":"ar"}'

You should receive a JSON response with an answer field containing an Arabic explanation grounded in the seeded documents, and a context array listing which chunks were retrieved.

Step 10: Evaluating Pipeline Quality

Three validation tests before shipping to production:

Alef normalization test. Index a document containing "إسلام" (with Hamza below Alef). Query for "اسلام" (plain Alef, no Hamza). With normalization enabled, recall should be 100%. Without it, recall is 0% — a hard failure invisible without this test.

Tashkeel roundtrip test. Index "الكِتَابُ" (with full Tashkeel). Query "الكتاب" (no diacritics, as most users type). The normalized forms must be identical — retrieval must succeed.

Symmetry canary test. Temporarily disable normalization in the query path only (keep it on in the indexer). If recall drops, your normalizers are already aligned. If recall stays identical, your corpus is already clean and normalization is a no-op — both fine outcomes, but you need to know which one you have.

Troubleshooting

"Cannot find package @lancedb/lancedb" — The package requires a compatible native binary for your OS/architecture. On Apple Silicon, install with npm install @lancedb/lancedb --os=linux is wrong — use plain npm install @lancedb/lancedb and let it download the correct darwin-arm64 binary automatically.

OpenAI 429 Rate Limited — The embedBatch function sends all texts in a single API call. For corpora with thousands of chunks, add a small delay between loop iterations: await new Promise((r) => setTimeout(r, 50)) after each batch.

Zero results from vector search — Confirm that the vector field in your documents is an array of 1536 numbers. If you passed a dimensions parameter to the OpenAI embedding call, the output size changes and LanceDB's stored schema will not match new searches.

LanceDB file permission error — The .lancedb directory is created at the process working directory. For Next.js on Vercel, use /tmp/.lancedb (writable in serverless functions) or point to an S3 bucket path.

Arabic text renders as question marks in logs — Use JSON.stringify(text) to inspect the raw Unicode values. Arabic characters in the range U+0600–U+06FF are valid UTF-8; rendering issues are usually a terminal font problem, not a data problem.

Next Steps

Now that your Arabic RAG pipeline is running, consider extending it with:

  • Hybrid search — Add a full-text search index on the normalizedText column using LanceDB's built-in FTS, then fuse BM25 and vector scores with Reciprocal Rank Fusion. The LanceDB tutorial covers this in depth.
  • Reranking — Pass the top 20 retrieved chunks through a cross-encoder reranker (Cohere rerank-multilingual-v3.0 supports Arabic natively) and send only the top 5 to Claude.
  • Arabic stop word filtering — Remove high-frequency function words (و، في، من، على، إلى، عن، مع) before embedding to reduce noise in retrieval.
  • Dialect support — For Egyptian, Gulf, or Levantine Arabic, extend normalizeArabic with dialect-specific substitution tables (alef-hamza omission patterns, idiomatic Tatweel usage).
  • Server-mode vector store — For multi-tenant production with high write concurrency, migrate from embedded LanceDB to Qdrant.

Conclusion

You have built a complete Arabic RAG pipeline in TypeScript: a Unicode normalization layer that resolves Alef variants, strips Tashkeel and Tatweel, and unifies Alef Maqsurah; a batched embedding pipeline using text-embedding-3-small; a LanceDB vector store with schema inference; a retrieval module that enforces normalization symmetry; and Claude Sonnet 5 for fluent Arabic answer generation.

The central invariant is normalization symmetry: the same rules, applied at index time and at query time. Break this and the pipeline degrades silently — every query returns results, but the right documents are missing.