ArabEval scores Arabic AI the way your users will judge it: native raters matched to each dialect, a healthcare specialist panel for clinical content, and a written rubric behind every score — delivered with a named expert's signature, not a platform ticket.
Illustrative scorecard — fictional values for demonstration only; not derived from a client project or a validated benchmark. One blended “Arabic score” hides exactly the failures your users will find. We report per dialect, per failure type — with the rubric and inter-rater agreement attached.
Frontier models are benchmarked in English and shipped in Arabic. Independent research keeps finding the same three cracks:
Model outputs scored against a written rubric — accuracy, dialect & register fit, instruction following, safety — by native raters matched to the dialect being judged.
Arabic medical and pharma content reviewed by a specialist panel from regulated healthcare — the review layer generic linguists cannot supply.
Adversarial testing in the Arabic users actually type: dialect slang, Arabizi, code-switching, and culturally loaded prompts that English test suites never generate.
A private, held-out Arabic benchmark written natively — not translated from English — so your team can measure every future release against ground truth.
Send us 300–500 of your real Arabic prompts and outputs. In one week you know — per dialect, per failure type — whether your Arabic is ready for the users you're selling to.
A retained expert panel on your release cadence — every model update graded before it ships, with the same rubric and the same raters.
For vendors and labs: ArabEval onboards as a named specialist supplier — Arabic evaluation and clinical review as a defined service line, NDA-first.
Launching Arabic to patients or the public? An independent expert gate before you ship: harm-graded findings and a signed readiness verdict your board can act on. Ships in days, not quarters.
A 30-minute call around your real Arabic outputs — not a demo set. We agree dialects, domains and risk lines.
The written rubric is drafted and agreed before a single item is scored. No moving goalposts, no vibes.
Every batch is scored twice by dialect-matched raters. Disagreements are adjudicated, never averaged away.
Scores, rubric, agreement statistics and the verdict memo — signed by the expert who stands behind them.
| ArabEval | Large data platforms | |
|---|---|---|
| Who rates your Arabic | Named, dialect-matched experts — and the lead evaluator signs the report | Vetted rater pools — skilled, but anonymous to you |
| Clinical content | A dedicated service line: specialist review from regulated healthcare, harm-graded P1–P3 | Handled inside general language operations |
| Dialect reporting | A scorecard per dialect family — MSA, Gulf, Egyptian, Levantine | Multi-language programmes, usually reported as one blended score per language |
| Price & speed | A published fixed-fee Sprint, delivered in 7 days | Custom enterprise pricing, quoted after a sales process |
| Best fit | Focused, high-stakes batches where every error is expensive | Very large programmes — millions of items, many languages |
| Neutrality | No model to sell — our only product is the verdict | Several also sell models, data platforms, or staffing |
Fifteen years in regulated healthcare and pharma — hospital specialist and senior medical delegate roles, leading portfolios that clinicians across the region rely on daily. Native Arabic across MSA and the Gulf, Egyptian and Levantine dialect families.
ArabEval exists because Arabic AI is being graded by people who would never catch the errors that matter — a wrong dialect register in a patient message, a dosage phrased one shade too casually, a safeguard that evaporates in Arabizi. We catch them, grade them, and sign for it.
Our own proof asset: 50 Arabic medical Q&A items, built against our written rubric to expose exactly the failures a blended score hides. We walk it through — methodology first — on every scoping call. Bring your hardest sample; we will grade it the same way, live.
NDA first, before any sample changes hands. Your prompts and outputs are seen only by the reviewers assigned to your project and are used only to evaluate them — never to train anything, never shared. Materials are retained only for the engagement and deleted on request, with deletion confirmed to you in writing. Your product's name is never used in our materials.
Initial evaluation coverage: Modern Standard Arabic, Gulf Arabic, Egyptian Arabic and Levantine Arabic — plus Arabizi and code-switched text in red-teaming. Other varieties are assessed for reviewer availability and suitability before a project is accepted; we never score a dialect we cannot staff natively.
The Sprint is seven days from kickoff at a fixed fee, published above. Retainer and supplier work runs on your release cadence, agreed in the scoping call — turnaround is a commitment we sign, not an estimate.
No. Commodity annotation is a race we refuse to enter. We sell expert judgement: evaluation, clinical review, red-teaming, and golden sets built to order — scored by named experts and signed.
Yes — ArabEval onboards as a specialist supplier for labs and language vendors, and as a subcontractor where a programme needs Arabic depth it cannot staff. One conversation starts it.
The usual start is a 30-minute scoping call: bring a sample of the Arabic output you need judged, and we return the rubric draft and a priced batch proposal.
Prefer email? hello@arabeval.com