The AI Benchmark for Payroll
EmBench Payroll measures whether AI models answer real payroll questions completely and correctly for the asker’s country. It covers 295 tasks across 8 countries, built from questions employees and payroll administrators ask. In-country payroll specialists reviewed and approved every answer key.
295
Tasks (held out, across 8 countries)
8
Countries
2,300
Criteria in total
29
In-country payroll specialists
33%
Of tasks fully solved by the best model (Claude Opus 5.5) in all three runs
Key takeaways
No model reliably answers payroll questions yet
The best model, Claude Opus 5.5, fully solves 33% of tasks in all three runs and 35% in none.
Payroll is local and dated
On the hard split the best model states only 75% of the state, province or local rules the keys require and 43% of the figures that changed in 2025 or 2026. Its fully-solved share spans 24 to 57% across the 8 countries.
Capability is not yet consistency
The best model fully solves 65.2% of tasks at least once in three runs, and only 33.3% in all three.
The best model meets 89.7% of specialist criteria but fully solves only 49% of tasks, mean of three runs.
| Segment | Share |
|---|---|
| Met in all three runs | 85% |
| Met in one or two runs | 9% |
| Never met | 6% |
Every headline-tier criterion (2,121) over three runs, Claude Opus 5.5. Mean criteria met per task: 89.7%.
Leaderboard
Each answer is checked against the criteria an in-country payroll specialist approved for that question. They cover the figures, conditions, filings and deadlines a complete answer must include for the asker’s country, plus statements it must not make.
The benchmark was built in three steps. First, questions were sampled from what employees and payroll administrators ask, then rewritten to stand alone with personal details removed. Second, an AI model drafted a reference answer and criteria for each question, and in-country payroll specialists corrected and approved them, with a source for every change. Third, each model answers every question three times without search or outside sources, and an AI grader checks each criterion separately. The held-out set has 295 tasks across 8 countries and 2,300 criteria.
Scores use the 270 tasks with fully reviewed keys. The other 25 are provisional until a qualified specialist reads them. Twelve more tasks are public as a sample; the held-out questions and keys are not released.
- Claude Opus 5.589.7
- Claude Sonnet 5.582.5
- Gemini 3.8 Flash78.9
- Pangaea73.6
- GPT-6 Luna70.7
- GPT-6 Sol67.0
- GLM 5.3 Flash66.7
- DeepSeek V4.1 Flash65.3
Criteria met is the share of criteria an answer passes; fully solved (Pass@1), the share of tasks with every criterion passed. Headline tier, mean of three runs per model. A rank is a separation only where a paired test by task, corrected across all 21 pairs, separates the two rows; the report lists every pair. GPT-6 Luna’s row covers 260 of the 270 tasks. An answer without a final response scores every criterion not met; a run lost to a failure outside the model is left out of that task’s average. Pangaea, Deel’s payroll model, is scored on one run, so it is not in the pair tests.
Cost per answer against criteria passed
Cost per answer against criteria passed
| Name | List-price cost per answer, log scale | Mean Criteria@3, headline tier |
|---|---|---|
| Claude Opus 5.5 | 0.0531 | 89.7% |
| Claude Sonnet 5.5 | 0.0250 | 82.5% |
| Gemini 3.8 Flash | 0.0084 | 78.9% |
| GPT-6 Luna | 0.0011 | 70.7% |
| GPT-6 Sol | 0.0085 | 67.0% |
| GLM 5.3 Flash | 0.0053 | 66.7% |
| DeepSeek V4.1 Flash | 0.0095 | 65.3% |
Top quadrant (List-price cost per answer, log scale 0.00050 to 0.00912, Mean Criteria@3, headline tier 74.3% to 100.0%): Gemini 3.8 Flash.
List price times tokens per answer, three-run mean; score is Mean Criteria@3, headline tier. Hollow markers: the platform returns no billed counts for DeepSeek V4.1 Flash and GLM 5.3 Flash, so their point is a floor from the visible answer length plus the reasoning tokens their records count. Shaded: high score, low cost. Each pill prints cost per answer, then criteria passed. One run of the whole set costs about $0.32 to $16 per model at list price. Prices and dates: report.
Payroll is local. So is model accuracy
| Model | GB n=37 | CA n=36 | DE n=38 | FR n=28 | ES n=35 | IN n=37 | PH n=23 | US n=36 |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 91%GB · 91% | 91%CA · 91% | 91%DE · 91% | 94%FR · 94% | 89%ES · 89% | 81%IN · 81% | 95%PH · 95% | 89%US · 89% |
| Claude Sonnet 5.5 | 86%GB · 86% | 83%CA · 83% | 84%DE · 84% | 89%FR · 89% | 76%ES · 76% | 68%IN · 68% | 93%PH · 93% | 87%US · 87% |
| Gemini 3.8 Flash | 82%GB · 82% | 79%CA · 79% | 76%DE · 76% | 82%FR · 82% | 75%ES · 75% | 74%IN · 74% | 87%PH · 87% | 80%US · 80% |
| GPT-6 Luna | 71%GB · 71% | 70%CA · 70% | 70%DE · 70% | 73%FR · 73% | 72%ES · 72% | 61%IN · 61% | 83%PH · 83% | 70%US · 70% |
| GPT-6 Sol | 68%GB · 68% | 66%CA · 66% | 66%DE · 66% | 70%FR · 70% | 67%ES · 67% | 60%IN · 60% | 76%PH · 76% | 67%US · 67% |
| GLM 5.3 Flash | 73%GB · 73% | 71%CA · 71% | 57%DE · 57% | 61%FR · 61% | 58%ES · 58% | 64%IN · 64% | 76%PH · 76% | 77%US · 77% |
| DeepSeek V4.1 Flash | 66%GB · 66% | 71%CA · 71% | 64%DE · 64% | 65%FR · 65% | 57%ES · 57% | 54%IN · 54% | 74%PH · 74% | 71%US · 71% |
Criteria passed, mean of three runs, on each country’s headline-tier keys. * FR has 28 of 37 keys in the headline tier and PH 23 of 38; the rest are provisional. Both country tables: report.
Where models break
The hard split is 113 tasks that no earlier-generation model fully solved in a single run. The chart shows criteria passed. On the split, the best model passes 82% of criteria. Fully solving a task is much rarer: the best model does it in all three runs on 8.0% of tasks, and in at least one run on 32.7%.
Criteria passed on the hard split, mean of three runs
±95% CI- Claude Opus 5.582.0% ±2.6
- Claude Sonnet 5.572.1% ±3.5
- Gemini 3.8 Flash70.5% ±2.9
- GPT-6 Luna60.6% ±2.9
- GPT-6 Sol58.6% ±2.8
- GLM 5.3 Flash56.5% ±4.3
- DeepSeek V4.1 Flash54.6% ±3.4
Headline tier, three runs per model, 95% intervals from a percentile bootstrap over tasks (10,000 resamples). A rank is a separation only where a paired test by task, corrected across all 21 pairs, separates the two rows; the report lists every pair. GPT-6 Luna’s row covers 260 of the 270 tasks. An answer without a final response scores every criterion not met; a run lost to a failure outside the model is left out of that task’s average.
Why models fail
Most misses are one required element: a cap, a condition, the person who files, a figure that changed in 2025 or 2026. Claude Opus 5.5 missed 154 criteria on the hard split; omission was the largest single type of miss (48%), as it is for every other row, and GLM 5.3 Flash also gives a wrong figure or rule in 25% of its misses.
One sunburst per model: causes inside, the five modes outside, the model’s missed criteria on the hard split in the centre. Pick a model above the chart; the cause cards below follow the pick. Verdicts are the grader’s on the first run; the sort is a model read.
| Cause | Mode | Share |
|---|---|---|
| Completeness | All | 48% |
| Completeness | Omission | 48% |
| Knowledge | All | 20% |
| Knowledge | Stale figures | 12% |
| Knowledge | Wrong statement | 7% |
| Knowledge | Wrong jurisdiction level | 1% |
| Grading | All | 31% |
| Grading | Wording | 31% |
| Total | 154 missed criteria | |
Omission 48%
The answer addresses the question but leaves out a required secondary fact.
Stale figures 12%
The answer gives a 2025 amount, or calls a 2026 rule proposed.
Wording 31%
The answer says the right thing in terms other than the key’s.
Wrong jurisdiction level 1%
The answer gives the national rule where a state, province or municipal rule applies.
Wrong statement 7%
The answer gives a different figure or rule from the key’s.
Share of the picked model’s missed criteria on the hard split, by cause.
Sample task
Task prompt
Public sample task · France · an employee on the company’s payroll · not from the test set
An employee in France covered by the national collective agreement for food-predominant retail and wholesale trade (IDCC 2216) asks how long maternity leave is. What is the statutory maternity leave duration in France, and what factors change it?
Specialist criteria, graded for Claude Opus 5.5
Key
- Country
- France
- Law as of
- 2026-09-28
- Criteria
- 6, each graded yes or no; 1 must-not-claim
Reference answer: the approved key the criteria were drawn from
Law: Under the Labour Code (L1225-17 and following), maternity leave for a first or second child is 16 weeks: 6 weeks before the due date and 10 weeks after the birth. When the employee already has at least two dependent children, or has already given birth to two viable children, it is 26 weeks: 8 before and 18 after. For twins it is 34 weeks (12 before and 22 after), and for triplets or more it is 46 weeks (24 before and 22 after). With a favourable medical opinion, up to 3 weeks of the leave before birth can be moved to after the birth. A doctor can add pathological leave for a pregnancy-related condition: up to 2 weeks before the birth and up to 4 weeks after. If the baby arrives early, the unused weeks before birth are added to the leave after birth, so the total stays the same. The employer may not employ the woman for a total of 8 weeks around the birth, including the 6 weeks after it. The CPAM pays daily benefits (IJSS) based on prior salary, capped by reference to the social security ceiling and subject to eligibility. There is no statutory obligation for the employer to top up pay. The time on leave counts as work for paid-leave accrual and seniority.
Criteria and verdict
States that maternity leave for a first or second child is 16 weeks: 6 weeks before the birth and 10 weeks after
Verdict Pass
Yes: the table gives 16 weeks for a first or second child, 6 before the birth and 10 after.
States that from the third child maternity leave is 26 weeks: 8 weeks before the birth and 18 after
Verdict Pass
Yes: the table gives 26 weeks, 8 before and 18 after, once two dependent children already exist.
States that maternity leave for twins is 34 weeks and for triplets or more is 46 weeks
Verdict Pass
Yes: the table gives 34 weeks (12 + 22) for twins and 46 weeks (24 + 22) for triplets or more.
States that the CPAM pays maternity daily benefits (IJSS), capped, and that the employer has no statutory obligation to top up pay
Verdict Miss
No: it names the CPAM daily allowances (IJSS), but not their cap or that the law requires no employer top-up.
States that the employer may not employ the woman for a total of 8 weeks around the birth, including 6 weeks after it
Verdict Pass
Yes: it cites article L1225-29, a ban on work for 8 weeks in total with 6 weeks after the birth.
Does not claim that an early birth shortens the total length of maternity leave
Verdict Pass
Yes: it says unused prenatal days move to the postnatal leave, so the total is unchanged.
5 of 6 criteria met, so criteria passed 83% and fully solved no. A criterion marked must-not-claim is met when the answer avoids the error.
The model’s answer
# Maternity Leave in France (Statutory Rules + IDCC 2216 Considerations) ## 1. Statutory duration (Code du travail, art. L1225-17 et seq.) The main factor is the **number of children expected and already in the household**. | Situation | Prenatal | Postnatal | **Total** | |---|---|---|---| | 1st or 2nd child | 6 weeks | 10 weeks | **16 weeks** | | 3rd child or more* | 8 weeks | 18 weeks | **26 weeks** | | Twins | 12 weeks | 22 weeks | **34 weeks** | | Triplets or more | 24 weeks | 22 weeks | **46 weeks** | \*The 26-week rule applies when the employee or her household already has at least 2 dependent children, or she has already given birth to at least 2 viable children. **Mandatory minimum:** The employer may not employ the woman for a total of **8 weeks**, including **6 weeks after the birth** (L1225-29), even if she waives part of the leave. ## 2. Factors that change the duration or its split **a) Voluntary redistribution (with a favourable medical opinion)** - **Any pregnancy:** Prenatal leave can be **reduced by up to 3 weeks**. Postnatal leave increases by the same amount. - If she is put on sick leave during that deferred period, the deferral is cancelled from the first day of sick leave. - **3rd child:** Prenatal leave can be **increased by up to 2 weeks** (10 + 16). - **Twins:** Prenatal leave can be **increased by up to 4 weeks** (16 + 18). **b) Pathological leave (congé pathologique)** - **Prenatal:** Up to **2 weeks (14 days)** on medical prescription, taken any time after the pregnancy is declared. CPAM pays it at the maternity daily-allowance rate. - **Postnatal:** Up to **4 weeks** for illness resulting from the delivery. This is paid as sickness, so waiting days and sickness rules apply. Dismissal protection still applies. **c) Timing of the birth** - **Early birth:** Unused prenatal days are carried over to postnatal leave. The total is unchanged. - **Very premature birth (more than 6 weeks before term) requiring hospitalisation:** Leave is extended by the number of days between the birth and the planned start of prenatal leave. - **Late birth:** Prenatal leave is extended up to the actual birth date. Postnatal leave is **not** reduced. **d) Child's hospitalisation** - If the child is still hospitalised **6 weeks after birth**, the mother may return to work. She can take the rest of her postnatal leave when the child leaves hospital. **e) Stillbirth or death of the child** - Maternity leave remains due if the birth occurs after **22 weeks of amenorrhoea** (or the child weighs at least 500 g). - Separate bereavement leave rights may also apply. **f) Death of the mother** - The father (or the mother's spouse/partner) may take the remaining postnatal leave. ## 3. IDCC 2216 points to check before answering the employee Many sector agreements improve on the statutory rules. For the food-retail/wholesale agreement, verify the **current consolidated text on Légifrance** and any company agreement for: - **Salary maintenance during maternity leave:** whether the employer tops up CPAM daily allowances (IJSS) to full or partial salary, and the **seniority condition** for this. - **Reduced working hours or breaks for pregnant employees** from a given month of pregnancy. - Any provisions on **breastfeeding time** or return-to-work arrangements. I'd recommend confirming these against the current text rather than relying on memory, as the agreement is amended periodically. ## 4. Payroll and admin reminders - **IJSS eligibility:** CPAM daily allowances require minimum affiliation and contribution/hours conditions (e.g., 10 months' affiliation at the expected birth date, plus hours or contribution thresholds). Submit the **attestation de salaire** via DSN. - **Subrogation:** If salary is maintained under the agreement or company policy, the employer may receive the IJSS directly. Check whether subrogation is in place. - **Rights preserved during leave:** - Maternity leave counts as **effective work** for **paid leave accrual** and **seniority**. - The employee is protected against dismissal during the leave and the following 10 weeks. - She is entitled to the **salary increases** granted during her absence (L1225-26). - **Legislative changes:** France is introducing a new **"congé de naissance"** (birth leave) via the 2026 Social Security financing law. It is designed to sit alongside maternity leave rather than replace it. Monitor the implementing decrees for start dates and pay rules. **Short answer for the employee:** Standard maternity leave is **16 weeks** (6 before and 10 after the birth). It is longer for a third child (26 weeks) and for multiple births (34 or 46 weeks). It can be extended for medical reasons (up to 2 extra prenatal weeks and 4 postnatal weeks) or for premature birth. The prenatal/postnatal split can be adjusted with a doctor's approval.
Methodology
- Tasks
- 295 held-out across all eight countries, 300 drafted (one dropped on a specialist’s verdict, four moved to the public sample set at the final freeze): 270 in the headline tier, 25 provisional; twelve public sample
- Countries
- 8: US, UK, CA, DE, FR, ES, IN, PH
- Experts
- 29 in-country payroll specialists on Deel’s payroll teams; of the 25 profiled, median 10 years in payroll. A read counts when the benchmark’s reader rules accept it as qualified. Profiles and rules: report
- Criteria per task
- 7.8 per task after the one-claim split, 2,300 in all: statements a complete answer must make, or claims it must not make
- Contamination
- Every task checked against a registry of public benchmark questions. 164 of the 300 tasks drafted as held-out share a governing statute with a task outside the held-out set, an upper bound; the top rows land within a few points either way on those tasks and on the rest
- Runs per model
- 3, closed book, no system prompt, 16,384 output tokens, temperature 0 where the provider honours it. A non-answer scores zero; a run that returns no response for reasons outside the model is left out of that task’s average
- Knowledge cut-offs
- Keys state the law as of 28 September 2026; vendor-stated cut-offs run from February to June 2026. The 84 criteria that name a 2025 or 2026 figure test that gap
- Grading
- GPT-6 Sol grades each criterion in its own call, blind to the model. On 245 validation criteria a three-judge panel agrees with the specialists at kappa 0.82 and GPT-6 Sol alone at 0.77; specialists agree with each other at 0.69. Judge checks: report
How tasks were built
Each task is a real question from an employee or payroll administrator, rewritten to stand alone with personal details removed. An AI model drafted each reference answer and its criteria from Deel’s internal payroll documentation. In-country specialists then approved, amended, or dropped each one, and cited a source for every change.
What EmBench Payroll does not measure
Running a live payroll, reconciling a payroll register, filing a tax return, or completing multi-step tasks in other software. Models answer each question with no search or outside sources, so a model’s knowledge of recent rule changes is part of what the score measures.
Acknowledgements
Thanks to Deel’s payroll specialists across 8 countries and the Deel AI team.
EmBench Payroll starts with the questions workers actually ask about their pay
“Most AI benchmarks test models on questions written for the test. EmBench Payroll starts with the questions workers actually ask about their pay, covering everything from why a tax figure changed this month to how a bonus is taxed and what happens to unused leave on departure.”
Frequently asked questions
A benchmark measuring whether AI systems can correctly answer real payroll questions, the kind payroll teams face daily across countries. Answers are graded point by point against references verified by in-country payroll specialists.
Tasks start from the questions and jobs Deel’s teams handle every day, with no personal or client data. In-country experts build and correct the answer keys. A judge model scores each answer against what a complete answer must include, checked against expert reviews. Each benchmark’s report has the full method.
Talk to us about the areas your model needs to get better at. We can evaluate it and build training data with the same experts and process.
No. Test questions and answers stay private, so scores reflect real skill, not memorization.
