Lab report: GPT‑5, Claude Opus 5 and Gemini on Arabic — who actually understands our dialects?
This is a lab report, not an opinion piece. BrandBoost AI (our bilingual marketing copilot) and Kaleem (speech therapy for children) depend entirely on language models, so we re-evaluate every quarter against a fixed tes…
This is a lab report, not an opinion piece. BrandBoost AI (our bilingual marketing copilot) and Kaleem (speech therapy for children) depend entirely on language models, so we re-evaluate every quarter against a fixed test set: 420 Arabic tasks spread across Modern Standard Arabic, Gulf and Egyptian dialects, with double human grading.
Methodology
- Understanding: extracting customer intent from real (anonymised) WhatsApp messages in different dialects.
- Generation: writing an Arabic ad in a specified tone, then matching it against the brand guide.
- Constrained summarisation: summarising a real-estate contract without dropping a single financial clause.
- Diacritics and pronunciation: generating fully vowelled sentences for children’s exercises — Kaleem’s hardest task.
Results in brief
- Claude Opus 5: most accurate on constrained summarisation and on holding a specified tone. Fewest diacritic errors. Occasionally over-cautious in short replies.
- GPT‑5: fastest at catching intent in dialect messages, especially Egyptian, and best at producing many ad variants quickly. Sometimes slips on Gulf legal terminology.
- Gemini (latest generation): strongest on multimodal tasks — reading an ad image and judging its language — and good in MSA, but weaker on fine-grained dialect.
There is no “best model”. There is the best model for this task, at this cost, at this latency.
What we actually changed in our products
In BrandBoost AI, first-pass generation now uses a fast model and the final brand-compliance review uses a more accurate one — two steps instead of one, at roughly 35% lower cost. In Kaleem, diacritics stay exclusively with the lowest-error model, because one mistake in a child’s pronunciation exercise costs all of the parents’ trust.
Three lessons specific to Arabic
- Dialect needs examples, not instructions. “Write in Gulf Arabic” is not enough; five real examples work wonders.
- Numbers and dates are a source of silent errors. We enforce one format and verify it programmatically after generation.
- RTL is not the model’s problem, it is your UI’s. Half the “errors” users reported were rendering, not text.
How to repeat this yourself
Start with fifty real tasks from your business, not from the internet. Grade them by hand once, then automate the re-run monthly. Models change faster than your product roadmap, and a fixed test set is the only thing that turns change into good news instead of a surprise.
