Arabic Text Normalization: Fix Search, RAG and Tokens

An Arabic-language product built by a competent team will still fail in ways its English equivalent never does. A user searches for الإسلام and gets nothing, even though the document says الاسلام. A RAG pipeline retrieves zero chunks for a question whose answer sits in paragraph two. The OpenAI bill for the Arabic tenant is 40% higher than the English one on the same traffic. A WHERE name = ? lookup fails on a name the user typed correctly.

These look like four unrelated bugs. They are one bug, and it lives in a layer most teams never write: text normalization.

The problem: one word, many byte sequences

Arabic has an orthography where several distinct Unicode code points render as visually identical or near-identical text, and native writers use them interchangeably. The word for "Islam" can legitimately be written الإسلام (with hamza under alef, U+0625) or الاسلام (bare alef, U+0627). Both are correct. Both are common. To a database, they are different strings.

The main offenders:

The last two rows are the ones that cause the 3am debugging session. Text extracted from a PDF often arrives in Arabic Presentation Forms-B, where each glyph is encoded in its positional shape rather than its base letter. It renders correctly on screen and compares false against everything.

The normalization pipeline

The fix is a pure function applied at exactly two points: when you index, and when you query. If those two ever diverge, you are back where you started.

The order matters. This is the sequence the research literature converged on, and the one used by AraToken, a 2026 paper on Arabic tokenization for Qwen3:

const HARAKAT = /[\u064B-\u065F\u0670]/g;      // fathatan..superscript alef
const TATWEEL = /\u0640/g;                      // kashida
const INVISIBLE = /[\u200B-\u200F\u202A-\u202E\u2066-\u2069]/g;
 
const LETTER_MAP: Record<string, string> = {
  'أ': 'ا', 'إ': 'ا', 'آ': 'ا', 'ٱ': 'ا',       // alef variants
  'ى': 'ي',                                       // alef maqsura
  'ة': 'ه',                                       // ta marbuta
  'ؤ': 'و', 'ئ': 'ي',                            // hamza carriers
};
 
export function normalizeArabic(input: string): string {
  return input
    .normalize('NFKC')                  // 1. fold presentation forms to base letters
    .replace(INVISIBLE, '')             // 2. drop ZWJ/ZWNJ/bidi controls
    .replace(TATWEEL, '')               // 3. drop kashida
    .replace(HARAKAT, '')               // 4. drop diacritics
    .replace(/[\u0623\u0625\u0622\u0671\u0649\u0629\u0624\u0626]/g, c => LETTER_MAP[c])  // 5. unify letters
    .replace(/[\u0660-\u0669]/g, d =>   // 6. Arabic-Indic digits
      String.fromCharCode(d.charCodeAt(0) - 0x0660 + 48))
    .replace(/[\u06F0-\u06F9]/g, d =>   // 7. Extended Arabic-Indic digits
      String.fromCharCode(d.charCodeAt(0) - 0x06F0 + 48))
    .replace(/\s+/g, ' ')
    .trim();
}

Step 1 is the one people skip, and it is the most important. NFKC folds the entire Presentation Forms block back to base letters in a single call — that alone fixes most PDF and OCR breakage before your own rules even run. Note that NFKC is deliberately lossy: it is a compatibility normalization, which is exactly what you want for a search key and exactly what you must not apply to text you intend to display.

Two columns, never one

The cardinal rule: normalize for matching, never for storage of the original.

ALTER TABLE documents
  ADD COLUMN title_norm text
  GENERATED ALWAYS AS (arabic_normalize(title)) STORED;
 
CREATE INDEX documents_title_norm_trgm
  ON documents USING gin (title_norm gin_trgm_ops);

The user sees title. The query hits title_norm. Diacritics on a Quranic verse, the exact hamza a poet chose, the spelling on a legal document — all preserved, all still searchable. Destroying the original to make search work is a trade nobody asks you to make.

In PostgreSQL you can implement arabic_normalize directly with translate() and regexp_replace(), or call out to a Rust or PL/Python extension. In Elasticsearch, the equivalent is an analyzer chain: icu_normalizer (nfkc_cf), then arabic_normalization, then decimal_digit, then optionally arabic_stem. Elasticsearch's built-in arabic analyzer covers part of this, but not the presentation-form folding — add icu_normalizer before it.

Why this decides whether your RAG works

Embedding models are more forgiving than exact match, but not forgiving enough. Vector search failure in Arabic is usually not a model quality problem — it is a chunking and normalization problem that shows up as a model quality problem.

Three things break specifically:

Chunk boundaries. Splitting on . and ? misses ؟ (U+061F) and ، (U+060C). Arabic prose also uses و as a coordinating prefix far more freely than English uses "and", so sentence-count heuristics tuned on English produce chunks that are wildly uneven. Split on the Arabic punctuation set explicitly.

Embedding drift. The same sentence with and without harakat produces measurably different vectors in most multilingual models. If your corpus mixes diacritized and undiacritized sources — very common when you combine classical texts with modern web content — you are embedding two dialects of the same language into different regions of the space. Normalize before embedding, and use identical normalization at query time.

Hybrid retrieval asymmetry. The BM25 leg of a hybrid retriever is exact-match and therefore fully exposed to every variant above. Teams tune the dense leg for weeks while the sparse leg silently contributes near-zero recall on Arabic.

A useful diagnostic: take 50 queries you know should hit, run them raw and normalized, and compare recall@10. If normalization alone moves that number by more than a few points, retrieval quality was never your bottleneck.

The token cost angle

Arabic is expensive on LLM APIs, and the reason is structural. Across model-native tokenizers, Arabic runs at roughly 2.4 tokens per word against 1.5 to 1.6 for English. Qwen2.5's byte-level BPE encodes Arabic at 2.18 tokens per word, about 1.4 times its English rate. That ratio propagates into everything: prompt cost, time-to-first-token, KV cache size, and how many concurrent Arabic sessions a fixed GPU budget can hold.

Normalization does not solve this, but it does reclaim a real slice of it. The AraToken work reports fertility dropping from 1.311 to 1.199 tokens per word with a normalization pipeline in front of a SentencePiece tokenizer — an 8.5% reduction, widening to roughly 18% between the best and worst configurations tested. The mechanism is simple: every stray tatweel, orphaned diacritic and presentation-form character is a token you paid for and the model gained nothing from.

Concretely, for a RAG system stuffing 8K tokens of Arabic context per request at scale, an 8% reduction is 8% off the largest line item in the bill, with no model change and no quality loss.

Two practical notes. First, normalize context but be careful with the user's own prompt — if they wrote diacritics deliberately, stripping them changes what they asked. Second, if you are fine-tuning or continuing pretraining on Arabic, normalize the training corpus and the inference path with the same function, or you will train on one distribution and serve another.

What to actually do this week

1. Write one normalizeArabic function. One file, no dependencies, exported from a shared package. Every service imports the same one. This is the whole architecture.
2. Add a normalized column or field alongside every Arabic text field you search on. Generated column in Postgres, .keyword-style subfield in Elasticsearch. Index that.
3. Normalize the query with the identical function. The single most common production bug in Arabic search is normalizing the index and forgetting the query.
4. Write the test table. الإسلام matches الاسلام. مـــرحبا matches مرحبا. مُحَمَّد matches محمد. فاطمه matches فاطمة. ٢٠٢٦ matches 2026. Presentation-form input matches base-form input. Six assertions catch nearly everything.
5. Measure recall before and after on real queries. You need the number to justify the work, and it is usually larger than anyone expects.

None of this is research. It is a hundred lines of unglamorous code that most Arabic-facing products simply never wrote — which is why "Arabic search is bad" reads as a fact of the language rather than what it is: an unimplemented layer.

If you are building Arabic-first products, the related pieces are worth reading too: why Arabic letters change shape explains the joining behaviour behind presentation forms, harakat and tashkeel covers what you are stripping and when you must not, and common Arabic RTL website mistakes handles the rendering side of the same problem.