The AI Benchmark for Global Work
EmBench Global Work measures whether AI systems can answer the questions global employment raises, across hiring, payroll, benefits, time off, and compliance in 150+ countries. Every answer is graded fact by fact against expert-verified references.
150+
Countries
601
Questions
4.6
Reference facts per question
75
Best score, by a frontier system with search
70
Pangaea, from memory alone, in 7.7 s median
Key takeaways
Memory is within five points of search
Pangaea scores 70 with no tools. The best frontier system with search scores 75, and the two tie on 52% of questions.
Memory is twice as fast
Pangaea answers in 7.7 s median, against 13.1 to 28.4 s for every system that searches.
Teaching beats tooling
The same 9-billion-parameter base model scores 45* with search tools and 70 after training on Deel’s knowledge.
The hard part is synthesis, not freshness
Questions needing two or more pages score 57, against about 72 for single-page questions.
A model that answers from memory ties the best search-equipped system on half the questions, in half the time.
| Segment | Share | Questions |
|---|---|---|
| Pangaea higher | 20% | 110 |
| Tie | 52% | 289 |
| Model with search scored higher | 29% | 162 |
Per-question scores compared on the questions both systems answered. Shares are rounded, so they sum to 101%.
Leaderboard
Eleven systems, the same 601 questions, one judge. Frontier systems answer with search tools over Deel’s knowledge base, the setting they’d use in production. Pangaea answers from memory alone. Score is the share of reference facts stated correctly; “necessary facts only” scores just the facts an answer is wrong without.
The two Base 9B rows are the starting model for Pangaea, Deel’s own model, shown for reference and not ranked. *Scored on 352 of the 601 questions, so not directly comparable.
- DeepSeek V4.1 Flash75
- Claude Opus 5.574
- GLM 5.3 Flash72
- Claude Sonnet 5.571
- Pangaea70
- Gemini 3.8 Flash69
- Claude Sonnet 568
- GPT-6 Sol59
- GPT-6 Luna52
- Base 9B + search45
- Base 9B19
Full table
| Rank | System · setting | Score | Necessary facts only | Median latency |
|---|---|---|---|---|
| 1 | DeepSeek V4.1 Flash with search | 75, best | 77 | 14.8 s |
| 2 | Claude Opus 5.5 with search | 74 | 79, best | 28.4 s |
| 3 | GLM 5.3 Flash with search | 72 | 75 | 26.7 s |
| 4 | Claude Sonnet 5.5 with search | 71 | 75 | 20.7 s |
| 5 | Pangaea from memory, no tools | 70 | 73 | 7.7 s |
| 6 | Gemini 3.8 Flash with search | 69 | 78 | 16.7 s |
| 7 | GPT-6 Sol with search | 59 | 67 | 18.5 s |
| 8 | GPT-6 Luna with search | 52 | 58 | 13.1 s |
| – | Base 9B + search* with search | 45 | 55 | 13.6 s |
| – | Base 9B from memory, no tools | 19 | 22 | 10.7 s |
Speed against score
Up and to the left is better: higher score, fewer seconds. Pangaea sits alone in the top left, within five points of the best system and about twice as fast as the nearest one.
Score plotted against median seconds per answer, same 601 questions. Pangaea scores 70 at 7.7 seconds, faster than every search-equipped system, which score 52 to 75 at 13 to 28 seconds.
| System | Median seconds per answer | Score, KB Benchmark |
|---|---|---|
| DeepSeek V4.1 Flash | 14.8 s | 75 |
| Claude Opus 5.5 | 28.4 s | 74 |
| GLM 5.3 Flash | 26.7 s | 72 |
| Claude Sonnet 5.5 | 20.7 s | 71 |
| Pangaea | 7.7 s | 70 |
| Gemini 3.8 Flash | 16.7 s | 69 |
| GPT-6 Sol | 18.5 s | 59 |
| GPT-6 Luna | 13.1 s | 52 |
| Base 9B + search* | 13.6 s | 45 |
| Base 9B | 10.7 s | 19 |
Fast and accurate (Median seconds per answer 0 s to 12 s, Score, KB Benchmark 65 to 80): Pangaea.
Best trade-off between speed and score: Pangaea, DeepSeek V4.1 Flash.
training on Deel’s knowledge: from Base 9B to Pangaea.
Median seconds for the complete answer in the evaluation harness. In the product, answers stream, so the first words appear within a second. Hollow markers are the base model before training. Dashed line: the best trade-off between speed and score. *Scored on 352 of the 601 questions.
Where models fail
Every system struggles most where an answer has to connect facts from more than one page. Knowledge going stale isn’t the problem.
57 vs 72
Multi-page synthesis is the clearest weakness
Questions needing facts from two or more pages score about 15 points below single-page questions for Pangaea. Connecting facts learned separately is harder than recalling one.
752 of 755
Freshness isn’t the problem
Of the 755 facts Pangaea missed in the latest run, 752 were stable: their pages hadn’t revised them recently. The gaps come from what the model learned, not from the knowledge base moving on.
81%
Problems show up in samples, not dashboards
One corpus build gave long chunks, which held 81% of the text, almost no restatements. A model can memorize text and still score zero on questions about it. We found it by reading output.
Sample task
Task prompt
- Brazil
- Termination
An employer wants to terminate a full-time employee in Brazil who has 3 years of service and is past probation. What notice period applies, and can it be paid in lieu?
How it’s scored
A low score points at the exact facts that were missed, which tells us where to look in the training data. That’s more useful than a single mark out of ten.
Reference facts and verdict
- Necessary
Notice is 30 days plus 3 additional days per year of service beyond the first
In the answer? Yes
- Necessary
Total notice is capped at 90 days
In the answer? Yes
- Necessary
Payment in lieu of notice is permitted
In the answer? Yes
- Supporting
Employees on working notice get reduced hours or paid days off
In the answer? No
Question score: 3 of 4 facts stated correctly.
Methodology
Questions are written to match what Deel’s operating teams ask every day. Each carries a reference answer broken into individual facts.
- Questions
- 601 across hiring, payroll, benefits, time off, and compliance
- Countries
- 150+ countries Deel operates in
- Reference facts
- 4.6 per question on average, each marked necessary (the answer is wrong without it) or supporting (useful, but optional). Verified by Deel domain specialists
- Grading
- A judge model checks each fact in turn: is it there, and is it stated correctly? A question’s score is the share of its facts the answer gets right
- Metric
- The KB Benchmark score, the share of reference facts stated correctly across all questions. EmBench Payroll uses its own metrics (criteria passed and fully solved over three runs), so scores don’t carry across the two benchmarks
- Systems and harness
- Frontier models run with search tools over the knowledge base, the configuration they’d use in production. Pangaea answers from memory with no tools. Latency is measured end to end on the same path
- Freshness control
- Every fact is tagged stable or volatile from its source page’s 90-day edit history, judged on whether an edit changed the value. Rewording or reformatting doesn’t count
- Refresh cadence
- About 15% of knowledge-base pages change every month. Questions are re-verified against current pages, and volatile facts are scored separately
Why trust these numbers
A benchmark is only as good as its reference answers. Every reference fact is verified by Deel domain specialists, the people who run payroll, compliance, and HR operations in the countries the questions cover. They check each fact against law and practice with sources, correct values, add facts the reference missed, and flag statements an answer must not make.
Checks on the grading
We audit the judge rather than assume it. Specialists review samples of its fact-level decisions, and this is how we caught and replaced judge versions that rewarded plausible-sounding but ungrounded answers.
Our model is on the board
Every system runs the same questions through the same harness and judge. Specialists set the reference facts, not the modeling team, and we publish where Pangaea loses.
Acknowledgements
Thanks to Deel’s domain specialists in payroll, compliance, and HR, and to the Deel AI team.
From benchmark to the real world
“Getting a global work question wrong can mean a missed notice period, an unlawful dismissal, or a hire who cannot legally start. EmBench Global Work is built to catch those errors in AI models. Its questions reflect the work Deel’s specialists resolve daily across hiring, payroll, and compliance for more than 40,000 organizations in over 150 countries.”
Frequently asked questions
A benchmark measuring whether AI systems can correctly answer real global-employment questions, the kind companies hiring across borders face daily. Answers are graded fact by fact against expert-verified references.
Tasks start from the questions and jobs Deel’s teams handle every day, with no personal or client data. In-country experts build and correct the answer keys. A judge model scores each answer against what a complete answer must include, checked against expert reviews. Each benchmark’s report has the full method.
Talk to us about the areas your model needs to get better at. We can evaluate it and build training data with the same experts and process.
No. Test questions and answers stay private, so scores reflect real skill, not memorization.
