How the 295 held-out payroll tasks were built and scored, and what the leading models get right and wrong.
EmBench Payroll is built from questions employees and payroll administrators asked Deel’s payroll support, not questions written for a test. In payroll, almost right means someone gets paid the wrong amount, so we score whether the whole answer is right. The best model we tested, Claude Opus 5.5, gets 90% of the details right but fully solves 49% of them, averaged over three runs. We hold every model to this bar, our own included: Pangaea Payroll, Deel’s own model, fully solves 19.3% and passes 73.6% of criteria.
Abstract. EmBench Payroll measures whether a language model answers real payroll questions completely and as the specialists’ key states, for the asker’s country. It is built in three steps. Real support questions become 300 de-identified, held-out tasks in eight countries, 295 of them in the final set. Claude Opus 5.5 drafts each key, a reference answer and about six binary criteria, later split to about eight; in-country payroll specialists read most keys before the freeze and correct them where needed, and keys no qualified reader has seen stay provisional. One LLM judge scores every criterion blind to the answering model, validated against how often two specialists agree. On the 270-key headline tier, closed book, the strongest model, Claude Opus 5.5, passes 90% of criteria but fully solves 49% of tasks, averaged over three runs. On the hard split, 113 tasks no previous-generation model fully solved, it fully solves 8% in all three runs and 33% at least once. Criteria passed there falls to 55 to 82%. A model sort of the best model’s misses on the hard split finds omissions of a required secondary fact first, then wording the grader did not accept, then figures that changed in 2025 or 2026. 25 keys stay provisional until a qualified specialist reads them.
1. Introduction
Payroll rules live in statute, collective agreements and official guidance. They change every tax year, and many are set by a state, a province or a municipality. A payroll answer helps only when it is complete: the right rule, this year’s figure, the exception that applies and who files. Missing one can still cost someone money.
EmBench Payroll asks whether an answer states everything a specialist in that country says it must, and nothing it must not claim. Each task is a question, a reference answer and a list of criteria. The model answers with no tools and no documents, and a judge model checks each criterion. A task is fully solved only when every criterion passes.
We lead with fully solved because people act on a payroll answer as a whole. An employee plans around it, and an administrator runs payroll from it. An answer that gives the rate but leaves out the cap, or the deadline but not who files, still leaves someone with the wrong pay or a missed filing. We report criteria passed to show how close a model gets, and fully solved to show how often its answer can be used as given.
The public sample task on maternity leave in France shows the gap. Claude Opus 5.5 gets every leave length right, but leaves out that the daily benefit is capped and that the employer does not have to top it up. That is 5 of 6 criteria passed, and the task is not fully solved.
It is for model builders who want a held-out payroll test, teams that deploy a model on payroll questions, and researchers who study model-drafted, expert-corrected keys.
What this version covers
295 held-out tasks, 35 to 38 in each of 8 countries: Canada, Germany, Spain, France, the United Kingdom, India, the Philippines and the United States. Questions come from employees and payroll administrators, on own payroll and through an employer of record. The law is as of 28 September 2026.
2. Task construction
Most of the work in EmBench Payroll went into the answer keys. Each task goes through the same seven steps, from a real question to a frozen key.
- Sample a real support question.
- De-identify it and rewrite it to stand alone.
- State who is asking; invent figures where needed.
- Draft the reference answer and criteria with Claude Opus 5.5.
- Have an in-country specialist correct the key.
- Apply accepted corrections and split compound criteria.
- Freeze the key as a hashed snapshot.
Where the questions come from
Every task starts from a question someone asked Deel’s payroll support in the six months before the build. Sampling per country from a pre-filtered pool keeps the mix close to what people ask. 600 items were drafted, 75 per country: 300 held-out, 30 development and 270 training. The scored held-out set holds 295 tasks: four of the 300 are in the public sample set and one was dropped on a specialist’s verdict. Items in one country that share a rule, a paraphrase or a dated change in the law land in the same split.
De-identification
Each question was rewritten so that no person, employer or contract can be identified. The drafting model saw only the de-identification scanner’s output. A check of every capitalised name before export found no names left to remove.
Who is asking
Each question opens with one neutral line naming the asker: an employee on own payroll, through an employer of record or with a co-employment provider, or an HR or payroll administrator.
Synthetic figures
13 of the held-out tasks carry constructed figures, invented in the shape of a worked case whose figures were private or missing. Two independent checkers recomputed 28 constructed items. A held-out item goes to adjudication when both dispute a figure a specialist approved.
Drafting
Claude Opus 5.5 drafted every reference answer and criterion from Deel’s internal payroll documentation. Each criterion is binary: a statement a complete answer must make, or a claim it must not make. 1,976 of 3,440 criteria on the 600 drafts were supported by the knowledge base, and specialists saw each criterion’s source state. A pass against each country’s dated change log edited 26 drafts and left 219 disagreements to the specialists. Every question cleared a check against public benchmark questions, once that check had failed on a planted one.
Specialist review
The readers are 29 payroll specialists on Deel’s own payroll teams, each reading the country they run payroll for. On the 25 profiles collected: median 10 years in payroll, from 4 to 26, 14 with ten years or more; prior employers 6 of 24 at a Big Four or accounting firm, 10 in-house payroll at a large employer, 8 a payroll provider or employer of record; 9 of 24 hold a payroll, HR or tax credential; 19 of 24 were nominated by their country’s payroll lead, the rest volunteered or were assigned by availability. Only aggregates are published; names and individual reads are not.
A key is in the headline tier when a qualified reader has read it: one who passed the seeded attention test, or a strict reader, whose correction reads marked under 90% of criteria met. 16 specialists passed the seeded test.
Each read asked whether the reference is correct and current, whether each criterion is correct and checks one thing, and for a verdict: approve, amend or drop. The specialist also graded one model answer, assigned at random across three families and stratified by country, so its effect on corrections can be tested.
An automatic key check sent flagged keys for a second read (appendix); judge validation is a separate task (chapter 3).
Correction lines and their confirmation
Of the 64 parsed key fixes, 35 were applied, whole or in part. Specialists wrote one fix per line in a fixed grammar, with a source for every edit and addition. Structured lines were taken as written, and a model restated prose as grammar lines. A model read each line and the benchmark lead decided. A line not accepted is not applied; its item is frozen as drafted and listed for adjudication. The model overruled a specialist once, and that item is held. Of two disagreeing reads, the one with an accepted correction wins; otherwise the key stands.
The one-claim split
The split raises criteria met slightly and changes fully solved little. Many drafted criteria bundled a rate and a cap, or a deadline and who files, so an answer stating three of four parts failed the whole criterion. Each such criterion was split into one claim per criterion in the parent’s own words, then checked for added or lost numbers and words. 540 of 694 proposed splits passed. The rest stay whole, as do 8 parents a specialist edited after the proposal. The keys hold 2,300 criteria, 7.8 per task, against about 5.7 as drafted.
Scored as parents again, the split run gives the best model 88.9% against 88.2% on the unsplit keys. On the 529 split parents all children pass 85% of the time, against 86% for the parents, so the grader is no more lenient per claim. The gain is partial credit: the mean child pass rate is 90%.
3. Scoring
Scoring 2,300 criteria for seven models, three times each, takes an automated grader. Before trusting its scores, we checked it against the specialists.
On 245 consensus labels in four countries, with disagreements dropped, the panel reaches kappa 0.82 and the scorer 0.77, against 0.69 between specialists; both are agreement figures, not accuracies. The first validation target is met. The stricter one is missed on recall for met criteria. The grader agrees with the specialists more often than the specialists agree with each other, and when it disagrees with them, it is the stricter of the two.
Judge design and settings
GPT-6 Sol, the evaluation platform’s built-in grader, scores every criterion on the leaderboard, one call per criterion. It sees the question, the answer and the criterion, never the answering model’s name. The answer is fenced as untrusted text, so instructions inside it cannot steer the grade. The judge is asked whether the answer states the criterion, never whether it is correct law. It scores the board for throughput; chapter 4 checks it against other judges and a reference-answer ceiling.
GPT-6 Sol is also ranked, so self-preference was measured on the second label set (below). On those labels no judge family favoured its own answers (the Opus-judge lift on the Claude rows’ fully solved, chapter 4, is the one sign the data give): the gap was -1.6 to +0.3 points, every interval crossing zero. Panel accuracy was 90.6% on Claude answers, 84.1% on Gemini answers and 87.2% on GPT answers.
Three judge families
The validation panel is Claude Opus 5.5, GPT-6 Sol and Gemini 3.1 Pro, majority per criterion; each judge is also reported alone.
The judge-validation study
Validation is a separate grading task, not the correction reads: 93.6% of those labels are “met”, so an always-“met” judge would score 93.6% on them. It has 12 validation and 2 qualification items per country, 112 in all, drawn from held-out tasks by a fixed rule and seed. Each carries one leading model’s answer, chosen before any read so that a third to a half of its criteria would fail. The 96 validation items carry 546 criteria, 39.6% expected to fail. The screen shows no reference and labels each criterion “The answer states that”. Each qualification answer was edited by hand, before upload, to omit one element: a figure, a date or a named form. These edited answers exist only inside the validation task. No scored run uses them, and no benchmark key was altered for the test.
Reader rules
The reader rules:
- A reader who marks both planted omissions as met is excluded from ground truth; their reads are kept for a sensitivity row.
- A reader with no failed criterion across the set is excluded the same way.
- No reader is excluded for disagreeing with the judge.
Every judge number uses these rules. Twelve readers are valid under them.
Agreement with specialists against the expert ceiling
Ground truth is the specialists’ consensus per criterion: the majority of three in Spain, two agreeing readers in Canada, the Philippines and the United States. That gives 245 labels, 68 not met. The 25 criteria where two readers disagreed are dropped, which removes the hardest cases and flatters every judge. Germany, France, the United Kingdom and India have one valid reader each and join at the second. The expert ceiling, Fleiss’ kappa among valid readers, is 0.69 on 270 criteria: Canada 0.80, Spain 0.63, the United States 0.62 and the Philippines 0.57, under the 0.6 floor.
| Judge | Agreement | Recall, met | Recall, not met | Precision, not met | Kappa (95% CI) |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 92.7 | 90.4 | 98.5 | 79.8 | 0.83 (0.75 to 0.90) |
| GPT-6 Sol (the scorer) | 89.8 | 85.9 | 100 | 73.1 | 0.77 (0.68 to 0.85) |
| Gemini 3.1 Pro | 91.8 | 88.7 | 100 | 77.3 | 0.81 (0.73 to 0.89) |
| Panel, majority of three | 92.2 | 89.3 | 100 | 78.2 | 0.82 (0.74 to 0.90) |
245 consensus labels on CA, ES, PH and US, 28% not met. Intervals are 2,000-draw item-clustered bootstraps.
Where judge and specialists differ, the judge fails what both readers passed: it is the stricter grader. Every judge recovers all, or all but one, of the agreed omissions. By country the panel is highest in Canada (0.94) and lowest in Spain (0.68), with the United States at 0.86 and the Philippines at 0.74; it does not track the specialists’ own agreement by country. No prompt was changed. On the reporting half, fixed in advance, the panel reaches 0.87 (0.74 to 0.95), with recall of 91.6% on met and 100% on failed criteria. On the tuning half it reaches 0.77.
The two bars
The first target asks that judge agreement is not significantly below expert agreement. It is met, and the panel is above every country’s pairwise kappa. The stricter target asks on the reporting half for kappa of at least 0.8 and a recall lower bound of at least 0.90 per class. On that half kappa is 0.87 and recall on failed criteria 100%; recall on met criteria, with a lower bound of 85%, does not clear it, so no judge clears the bar.
A second label set
The judge was also measured against a second label set: the correction reads of the 10 of 24 readers who marked under 90% of criteria met, 1,440 criteria in six countries. The panel’s kappa was 0.61 (0.54 to 0.67), and 0.77 on held-out tasks, recovering 87% of the specialists’ failed marks. It covers more countries and a harder label set, and shows one of the judge’s measured weaknesses. On “must not claim” criteria, 2% of labels, it reads a hedged claim as no claim, and caught 3 of the 28 the specialists failed.
4. Results
The best model gets most details right, but about half of its answers still leave out or misstate something a specialist requires.
Values are the final keys with the one-claim split, frozen on 16 October; every run on 16 October.
Setup
Seven models answer every task three times. Seven models are ranked in this version; the lineup grows as more are run. The runs, on 16 October 2026, used the platform defaults: temperature 0 where accepted (Claude models take none), top-p 0.7, top-k 50, 16,384 output tokens, provider-default reasoning effort, no system prompt and no retrieval. The judge is GPT-6 Sol at the platform’s default settings, one call per criterion, and failed requests were not re-sent. 73 of the 270 headline tasks (27%) were fully solved by no model in any run.
An answer that ends without a final response inside the output budget counts as not answered and scores zero; a run that returns no response for reasons outside the model is left out of that task’s average. An empty answer would score 17.1% on criteria passed, from the 16.5% of headline-tier criteria that are must-not-claim and are met by claiming nothing, and 0% fully solved. Of 810 task-runs per model on the headline tier, this applied to 97 for GLM 5.3 Flash, 66 for DeepSeek V4.1 Flash, 0 for GPT-6 Luna and 0 for GPT-6 Sol, and to 0, 0 and 0 for Claude Opus 5.5, Claude Sonnet 5.5 and Gemini 3.8 Flash. Each row’s averages run over the task-runs that returned an answer: GPT-6 Luna over 736 of 810, GLM 5.3 Flash over 807, GPT-6 Sol over 809, and every other row over all 810. A task every run left out is absent from that row; where one run was left out, fully solved and solved-at-least-once count the task’s answered runs.
Criteria passed (Mean Criteria@3) is the share of criteria passed, averaged over runs and tasks. Fully solved (Pass@1) is the share of tasks with every criterion passed, averaged over the three runs; it is the leaderboard number. Fully solved in all three runs (Pass^3) is the consistency floor, and solved at least once (Pass@3) is the ceiling. Two rows are ordered only when a paired test by task, corrected across all 21 pairs, separates them; otherwise they are a statistical tie.
Headline lineup
On criteria passed the top three rows are separated from each other and from every row below. The four rows below them are close. On fully solved, GLM 5.3 Flash is ahead of the other three, GPT-6 Luna and DeepSeek V4.1 Flash are both ahead of GPT-6 Sol, and GPT-6 Luna and DeepSeek V4.1 Flash are a statistical tie. On criteria passed, GPT-6 Luna is ahead of DeepSeek V4.1 Flash and GPT-6 Sol. The appendix lists each test.
| Model | Fully solved, avg. of 3 runs ±95% CI | Fully solved in all 3 runs ±95% CI | Solved at least once | Criteria passed ±95% CI |
|---|---|---|---|---|
| Claude Opus 5.5 | 48.9% ±5.2 | 33.3% ±5.5 | 65.2% | 89.7% ±1.4 |
| Claude Sonnet 5.5 | 33.1% ±4.8 | 20.0% ±4.7 | 48.1% | 82.5% ±2.1 |
| Gemini 3.8 Flash | 24.3% ±4.6 | 15.6% ±4.2 | 34.1% | 78.9% ±2.0 |
| Pangaea Payroll, one run | 19.3% ±4.6 | n/a (one run) | n/a (one run) | 73.6% ±2.8 |
| GLM 5.3 Flash | 18.1% ±3.7 | 6.7% ±2.9 | 32.2% | 66.7% ±3.0 |
| GPT-6 Luna | 12.2% ±3.4 | 6.9% ±3.1 | 18.5% | 70.7% ±2.1 |
| DeepSeek V4.1 Flash | 11.9% ±3.1 | 4.4% ±2.4 | 20.7% | 64.7% ±2.6 |
| GPT-6 Sol | 7.0% ±2.6 | 3.3% ±2.0 | 11.9% | 67.0% ±2.1 |
Headline tier, 270 keys. Pangaea Payroll is from one run: its answers were generated outside the benchmark runner and scored on the final keys with the board’s grader, so it has no three-run columns and is not part of the paired tests.
Claude Sonnet 5.5 and Gemini 3.8 Flash are the closest pair in the top three. They separate on criteria passed and on fully solved, but not on fully solved in all three runs.
Criteria passed sits near its ceiling and barely moves between runs; fully solved moves a lot. On the 270 headline tasks the best model fully solves 65% at least once but 33% in all three runs. Re-grading one run on all 295 tasks changed about 1% of verdicts and flipped 13 of 295 fully-solved verdicts. The other flips, about one task in four for the best model, are the model answering differently. A single run carries about five points of run noise beyond its sampling interval, so the leaderboard uses three.
GPT-6 Sol writes a third of the words the top three rows write, a mean of 184 against 625, and answers the headline question only; every lower row shows a wide gap between its two columns. Cost does not explain the order. At list price the 295 tasks cost about $0.32 to $16 per model per run, $0.0011 to $0.0531 per answer; the DeepSeek V4.1 Flash and GLM 5.3 Flash figures are floors from visible answer length, as the platform returns no billed counts for those providers.
Per answer, mean of three runs, as words, billed output tokens and list-price cost: Claude Opus 5.5 701, 2,633, $0.0531; Claude Sonnet 5.5 541, 2,482, $0.0250; Gemini 3.8 Flash 625, 2,224, $0.0084; GPT-6 Sol 184, 833, $0.0085. Billed output is three to six times the visible text where the provider counts thinking inside it.
The EmBench Payroll page plots cost and billed output tokens per answer against criteria passed for every model.
Run-to-run consistency
Criteria passed barely moves between runs; fully solved does. The same model, tasks and settings, run three times, on all 295 tasks, provisional keys included; a run without a final answer scores every criterion not met. Task-bootstrap intervals in brackets.
| Model | Criteria passed, run 1 | Criteria passed, avg. of 3 runs | Fully solved, run 1 | Solved at least once | Fully solved in all 3 runs |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 89.8% | 90.0% (88.6 to 91.3) | 47.1% | 66.4% (61.0 to 71.9) | 34.6% (29.2 to 40.0) |
| Claude Sonnet 5.5 | 82.6% | 82.6% (80.5 to 84.5) | 32.9% | 48.1% (42.4 to 53.9) | 20.3% (15.9 to 25.1) |
| Gemini 3.8 Flash | 79.7% | 79.1% (77.2 to 81.0) | 25.8% | 34.9% (29.5 to 40.3) | 15.3% (11.2 to 19.3) |
| GLM 5.3 Flash | 65.9% | 65.9% (63.0 to 68.6) | 17.0% | 31.2% (26.1 to 36.6) | 6.1% (3.4 to 8.8) |
| DeepSeek V4.1 Flash | 65.4% | 64.8% (62.3 to 67.1) | 12.5% | 20.7% (15.9 to 25.4) | 4.7% (2.4 to 7.1) |
| GPT-6 Luna | 71.1% | 71.1% (69.1 to 73.1) | 12.4% | 19.4% (14.8 to 23.9) | 7.0% (4.2 to 10.2) |
| GPT-6 Sol | 66.9% | 67.2% (65.3 to 69.2) | 6.8% | 11.9% (8.1 to 15.6) | 4.1% (2.0 to 6.4) |
Rows cover all tasks; a run without a final answer counts as not solved. For the top rows a flip is usually a single criterion, for the lower rows more often two or more. Most flips come from the model answering differently; the grader accounts for a small part. A single-run fully-solved figure carries about five points of run noise beyond its sampling interval. The hard split, cut from single filter runs, carries the same noise.
The EmBench Payroll page plots each model’s share of tasks solved at least once against its share solved in all three runs. The gap between the two is what a model gets right only some of the time.
Gross headroom counts, for each pair of models, the tasks the first fully solved at least once in three runs that the second never fully solved. The EmBench Payroll page shows it for every pair, on the headline tier.
By country
On the final values the best model leads everywhere; on criteria passed India is lowest for six of the seven models (GLM 5.3 Flash is lowest in Germany), and all its headline keys rest on strict readers’ reads. Criteria passed holds the best model in a band of 81 to 95% across countries; on fully solved it spans 24 to 57%. The Philippines carries 15 provisional keys beside its 23 headline keys. Counted, they would lower its criteria passed from 95.2 to 94.2% for the best model and from 75.9 to 73.9% for GPT-6 Sol, three-run means, and change no order. They stay out because the reads on them do not verify: each was read three times by readers the seeded test excludes, and on the 23 PH keys read by both a qualified and a set-aside reader, the set-aside readers approved 15 of the 16 reads on keys the qualified reader amended and caught 0 of 5 planted omissions.
Criteria passed, mean of three runs, by country.
| Model | GB, n=37 | CA, n=36 | DE, n=38 | FR*, n=28 | ES, n=35 | IN, n=37 | PH, n=23 | US, n=36 |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 91% | 91% | 91% | 94% | 89% | 81% | 95% | 89% |
| Claude Sonnet 5.5 | 86% | 83% | 84% | 89% | 76% | 68% | 93% | 87% |
| Gemini 3.8 Flash | 82% | 79% | 76% | 82% | 75% | 74% | 87% | 80% |
| GLM 5.3 Flash | 73% | 71% | 57% | 61% | 58% | 64% | 76% | 77% |
| DeepSeek V4.1 Flash | 66% | 71% | 64% | 65% | 57% | 54% | 74% | 71% |
| GPT-6 Luna | 71% | 70% | 70% | 73% | 72% | 61% | 83% | 70% |
| GPT-6 Sol | 68% | 66% | 66% | 70% | 67% | 60% | 76% | 67% |
Criteria passed, average of three runs, on each country’s headline-tier keys. * FR: 28 of 37 keys are in the headline tier; PH 23 of 38.
Fully solved in all three runs, by country. The best model in each country is marked.
| Model | GB, n=37 | CA, n=36 | DE, n=38 | FR*, n=28 | ES, n=35 | IN, n=37 | PH, n=23 | US, n=36 |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 38%, best | 31%, best | 29%, best | 32%, best | 29%, best | 24%, best | 57%, best | 36%, best |
| Claude Sonnet 5.5 | 24% | 11% | 16% | 21% | 11% | 16% | 35% | 31% |
| Gemini 3.8 Flash | 14% | 17% | 13% | 18% | 11% | 16% | 26% | 14% |
| GLM 5.3 Flash | 3% | 8% | 3% | 4% | 6% | 8% | 9% | 14% |
| DeepSeek V4.1 Flash | 5% | 11% | 0% | 0% | 6% | 0% | 9% | 6% |
| GPT-6 Luna | 6% | 6% | 3% | 11% | 3% | 3% | 27% | 6% |
| GPT-6 Sol | 0% | 6% | 0% | 7% | 3% | 5% | 9% | 0% |
The hard split
The hard split holds 113 tasks, 105 in the headline tier, that no previous-generation model fully solved. On criteria passed the successors keep their full-set order on it; that shows the filter did not reorder the families, not that it had no effect. On fully solved four rows are at zero, so that column cannot order them. DeepSeek V4.1 Flash and GLM 5.3 Flash have no previous generation and are reported on a split they did not help cut. The split is re-cut at every freeze.
| Model | Fully solved in all 3 runs ±95% CI · at least once | Criteria passed, mean of 3 runs ±95% CI |
|---|---|---|
| Claude Opus 5.5 | 9 of 113, 8.0% ±4.9 · 32.7% | 82.0% ±2.6 |
| Claude Sonnet 5.5 | 3 of 113, 2.7% ±3.1 · 15.9% | 72.1% ±3.5 |
| Gemini 3.8 Flash | 2 of 113, 1.8% ±2.2 · 7.1% | 70.5% ±2.9 |
| GLM 5.3 Flash | 0 of 113, 0.0% (0 to 3.2, exact) · 10.6% | 56.5% ±4.3 |
| DeepSeek V4.1 Flash | 0 of 113, 0.0% (0 to 3.2, exact) · 2.7% | 54.6% ±3.4 |
| GPT-6 Luna | 0 of 105, 0.0% (0 to 3.5, exact) · 1.0% | 60.6% ±2.9 |
| GPT-6 Sol | 0 of 113, 0.0% (0 to 3.2, exact) · 0.9% | 58.6% ±2.8 |
Fully solved is the count of split tasks solved in all three runs, out of the tasks the model answered in all three, with its share and interval; a row with no task solved in all three runs carries the exact binomial 95% interval in place of the bootstrap, which has zero width on a zero count. The share solved at least once follows the dot.
Each filter model ran once at leaderboard settings. Claude Opus 5 fully solved 153 of 295 tasks, Gemini 3.7 Flash 68, GPT-5.6 Luna 53, GPT-5.6 Sol 63 (41 tasks returned no answer and count as not solved) and Claude Sonnet 5 41. Filtering on each family’s predecessor partly confounds comparisons between families.
Failure modes
Omission is 48% of the best model’s misses on the hard split: the question answered, a required secondary fact left out. Stale figures, a 2025 amount or a 2026 rule called “proposed”, are 12%. Wording, the right point in other terms, is 31%; that is grader strictness. The national rule where a state, province or municipal rule applies is 1%. A wrong figure or rule, including a few keys since corrected, is 7%.
A model reader sorted each missed criterion by cause, against the answer, under one rubric, first run, frozen keys: 154 criteria for the best model. It is a model read, not a specialist read. Omission is the largest single cause for every other row too; GLM 5.3 Flash also gives a wrong figure or rule in 25% of its misses. Claude Sonnet 5.5 misses 250 criteria (omission 46%), Gemini 3.8 Flash 268 (60%, stale figures 16%), GPT-6 Sol 377 (69%), GLM 5.3 Flash 263 (39%, a wrong figure or rule 25%), DeepSeek V4.1 Flash 324 (54%) and GPT-6 Luna 321 (64%).
Typing the criteria by their text gives the same picture. On the hard split the best model states 43% of criteria naming a figure that changed in 2025 or 2026, 75% of state, province or local rules, 88% of deadlines and 82% of rates and amounts. The stated cut-offs end three to eight months before the reference date, and the 84 criteria that name a 2025 or 2026 figure are where that shows.
Each criterion gets one type from its text; the table gives the share the best model states on the hard split (frozen keys, first run).
| Criterion type | Criteria on the split | Claude Opus 5.5 states |
|---|---|---|
| A figure that changed in 2025 or 2026 | 42 | 43% |
| A state, province or local rule | 110 | 75% |
| A deadline or notice period | 65 | 88% |
| A rate, amount or count | 310 | 82% |
| A form, filing or authority | 29 | 83% |
Other checks
Shown answer. Two of the three answer families show no detectable difference where their own answer was shown to the specialist; GPT-6 Sol scores higher where its answer was shown. On the keys as drafted every interval included zero; on the final keys, criteria met, shown minus not shown, three runs on the final keys: Opus 5.5 -0.1 (-3.3 to +2.9), Gemini 3.8 Flash -2.9 (-7.1 to +1.3), GPT-6 Sol +4.9 (+0.5 to +9.2) points on 259 items with a random assignment; only GPT-6 Sol’s interval excludes zero, and it is the family whose answers were shown least often corrected.
Rule overlap. Of the 300 tasks drafted as held-out, 164 share a governing statute with a training task, an upper bound, and 136 share none. The top rows land within a few points either way on the two groups.
Checks on the grading
Judge ceiling. Measured on the final keys, the frozen reference answers echoed twice through the platform grader score 98.7% of criteria and 91.0% fully solved (87.7 to 94.4) on the headline tier, over the 268 headline tasks whose echo matched the reference in both runs. Even the reference answers fully solve only 91.0% of tasks, so fully solved on the board is read against that ceiling.
Judge sensitivity. Among the four rows re-graded, the order is the same under all three judges on both metrics. The stored first-run answers of Claude Opus 5.5, Claude Sonnet 5.5, Gemini 3.8 Flash and GPT-6 Sol were re-graded by the Opus 5.5 and Gemini 3.8 Flash judges and again by GPT-6 Sol, on split keys and the headline tier. The Opus 5.5 judge lifts criteria met by +1.3 to +3.3 points and fully solved by +2.6 to +7.8. The Gemini 3.8 Flash judge is stricter: -2.3 to -3.7 and -2.6 to -8.5 points. Under the Opus 5.5 judge the two Claude rows gain most on fully solved (+6.7 and +7.8 points against +2.6 and +4.4). That is the one sign of same-family leniency in the data, and it changes no order. A three-judge majority moves a row by 0.1 to +0.4 criteria points. A second GPT-6 Sol pass flips 1.0 to 1.7% of criteria and 1 to 13 fully-solved tasks of 270.
Task-level agreement. Of 48 validation answers with consensus labels, the specialists call 12 fully solved. The scorer confirms 7 and passes none they fail (kappa 0.68).
Grader strictness on reference and specialist text. On text a person wrote, the grader withholds credit on about 0.6% of stated points. On the echoed reference answers it withholds credit on 10 of 2,305 criteria (0.4%), none of them “must not claim” criteria. On the blind answers the grader misses 3 of the 45 points they make.
Blind re-solve pilot. Twelve specialist answers written without the key, each giving the headline answer in short form, fail 45 of 87 criteria: 41 are omissions of secondary points the short answers did not attempt, 3 grader wording misses and 1 contradiction under adjudication. A full re-solve will run in the next version.
5. Limitations
The results above should be read with these limits in mind.
One read per key
Most keys rest on one specialist read, and one reader can miss a wrong criterion. The key check finds inconsistent keys and recompute conflicts. It cannot find a key that is wrong but consistent, and most approved keys carried no signal. The blind re-solve pilot found one contradiction in 87 criteria (chapter 4). The next version will move to three reads per key.
Model-drafted keys
The keys were drafted by a model and corrected by experts. The leading rubric benchmarks have expert-written keys. A drafter’s error survives wherever a specialist approved without checking. Before review, about a quarter of the drafts carried a criterion the internal knowledge base contradicted or lacked.
Provisional keys
25 keys are provisional: France 9, the Philippines 15 and India 1. They are out of every headline aggregate, unevenly: France prints on 28 of 37 keys and the Philippines on 23 of 38. A country’s headline number covers the keys a qualified reader saw, which may not be a random sample of that country.
Single filter runs
Each filter model ran once, so another draw would move some tasks in or out of the hard split.
The judge ceiling
Specialist agreement is moderate, 0.69, and under 0.6 in the Philippines, measured on four countries. The scorer is the weakest judge at 0.77, misses the stricter recall bar for met criteria and reads hedged negative claims as no claim. A change of judge moves fully solved more than criteria passed, and fully solved has a ceiling of 91.0% (chapter 4).
The drafter effect
Claude Opus 5.5 drafted every key, and Claude models are ranked. The headline board is not adjusted.
What it does not measure
EmBench Payroll does not measure running a live payroll, filing a tax return, or multi-step work in other software. Models answer with no search or outside sources, so knowing recent rule changes is part of what the score measures.
Room to improve
Of the 270 headline tasks, 73 (27%) were never fully solved by any of the seven models, in any run. Our grader is strict: even the reference answers fully solve only 91% of tasks through it. That still leaves room for every model to improve. If you build models for operations work, test them on the questions people actually ask, and score the whole answer.
6. Dataset card
Version and dates
v1.0: frozen 16 October 2026, one-claim split applied; 295 tasks, 2,300 criteria. Models run on 16 October 2026.
Reference date: law as of 28 September 2026. Launch: 20 October 2026.
Counts per country and tier
| Country | Tasks | Headline | of which strict reader | Provisional |
|---|---|---|---|---|
| Canada | 36 | 36 | 0 | 0 |
| Germany | 38 | 38 | 0 | 0 |
| Spain | 35 | 35 | 0 | 0 |
| France | 37 | 28 | 28 | 9 |
| United Kingdom | 37 | 37 | 0 | 0 |
| India | 38 | 37 | 37 | 1 |
| Philippines | 38 | 23 | 0 | 15 |
| United States | 36 | 36 | 1 | 0 |
| All | 295 | 270 | 66 | 25 |
Frozen 16 October 2026. Of the 270 headline keys, 204 rest on a seeded-test passer’s read and the rest on a qualified reader’s read under the verification rules. The 25 provisional keys were read only by readers who failed or did not take the test, or not at all.
Reader and exclusion rules
29 specialists read keys (25 profiled: median 10 years in payroll); 16 readers passed the seeded test of 14 items per country. Passers and strict readers verify keys. Judge ground truth excludes readers who mark both planted omissions as met or fail no criterion, never readers who disagree with the judge. Names and individual reads are not released.
Judge statistics
Scorer GPT-6 Sol. Validation: panel kappa 0.82 (0.74 to 0.90) on 245 labels in four countries, reporting half 0.87, ceiling 0.69, scorer 0.77; judges with each other 0.89 to 0.95. Second label set: 0.61 on 1,440 strict labels, 0.77 held-out. First target met, stricter target not. DE, FR, GB and IN join at their second valid read.
Sample set
The sample set holds twelve tasks: eight built and reviewed like the held-out set and drawn from the training half, and four drawn from the held-out set, each published with a specialist’s answer written without sight of the key. A specialist who passed the seeded attention test approved each sample key as drafted. Each leaderboard model answered the twelve once on the platform; the page’s sample task comes from those runs, never from the held-out set. The sample set will be published on Hugging Face; until then it is available on request. Held-out questions, reference answers and criteria are not released; a submitted model is run on the held-out tasks with the benchmark judge and settings.
Changelog
- v1.0, 16 October 2026. One task dropped on a specialist’s verdict and four moved to the public sample set with their specialists’ blind answers; 295 held-out tasks, 2,300 criteria, 270 in the headline tier.
7. Appendix
Bootstrap method
Intervals are 95% percentile bootstraps over tasks, 10,000 resamples. Each task first gets its values: criteria share over three runs, and whether it was fully solved in all three or in at least one. Model differences use the paired per-task difference, so task difficulty cancels. Leaderboard tests are paired by task with Benjamini-Hochberg correction over all 21 pairs at 5%. Resampling by statutory rule (178 rule keys over the 295 tasks) widens the best model’s interval on criteria passed from ±1.4 to ±1.8 and on fully solved from ±5.5 to ±6.0. The task-level interval is printed; the paired tests are by task only.
Paired tests
Paired tests by task on the 260 headline tasks every model answered in all three runs, Benjamini-Hochberg correction over all 21 pairs at 5%. Chapter 4 summarises them.
| Metric | Pair | p | Result |
|---|---|---|---|
| Criteria passed | Claude Opus 5.5 over Claude Sonnet 5.5 | 5.3 × 10⁻¹⁵ | Separated |
| Criteria passed | Claude Opus 5.5 over Gemini 3.8 Flash | 1.1 × 10⁻²⁸ | Separated |
| Criteria passed | Claude Opus 5.5 over GPT-6 Sol | 7.0 × 10⁻⁶² | Separated |
| Criteria passed | Claude Opus 5.5 over DeepSeek V4.1 Flash | 4.7 × 10⁻⁵⁵ | Separated |
| Criteria passed | Claude Opus 5.5 over GLM 5.3 Flash | 1.7 × 10⁻³⁹ | Separated |
| Criteria passed | Claude Opus 5.5 over GPT-6 Luna | 2.4 × 10⁻⁵³ | Separated |
| Criteria passed | Claude Sonnet 5.5 over Gemini 3.8 Flash | 7.7 × 10⁻⁵ | Separated |
| Criteria passed | Claude Sonnet 5.5 over GPT-6 Sol | 1.8 × 10⁻²⁹ | Separated |
| Criteria passed | Claude Sonnet 5.5 over DeepSeek V4.1 Flash | 8.8 × 10⁻³⁷ | Separated |
| Criteria passed | Claude Sonnet 5.5 over GLM 5.3 Flash | 1.8 × 10⁻²⁴ | Separated |
| Criteria passed | Claude Sonnet 5.5 over GPT-6 Luna | 9.9 × 10⁻²⁴ | Separated |
| Criteria passed | Gemini 3.8 Flash over GPT-6 Sol | 4.9 × 10⁻²⁴ | Separated |
| Criteria passed | Gemini 3.8 Flash over DeepSeek V4.1 Flash | 1.1 × 10⁻²⁰ | Separated |
| Criteria passed | Gemini 3.8 Flash over GLM 5.3 Flash | 2.3 × 10⁻¹³ | Separated |
| Criteria passed | Gemini 3.8 Flash over GPT-6 Luna | 3.1 × 10⁻¹³ | Separated |
| Criteria passed | GPT-6 Sol and DeepSeek V4.1 Flash | 0.17 | Tie |
| Criteria passed | GPT-6 Sol and GLM 5.3 Flash | 0.76 | Tie |
| Criteria passed | GPT-6 Luna over GPT-6 Sol | 2.8 × 10⁻⁶ | Separated |
| Criteria passed | DeepSeek V4.1 Flash and GLM 5.3 Flash | 0.13 | Tie |
| Criteria passed | GPT-6 Luna over DeepSeek V4.1 Flash | 8.9 × 10⁻⁵ | Separated |
| Criteria passed | GLM 5.3 Flash and GPT-6 Luna | 0.071 | Tie |
| Fully solved | Claude Opus 5.5 over Claude Sonnet 5.5 | 1.6 × 10⁻¹¹ | Separated |
| Fully solved | Claude Opus 5.5 over Gemini 3.8 Flash | 6.8 × 10⁻²⁰ | Separated |
| Fully solved | Claude Opus 5.5 over GPT-6 Sol | 2.2 × 10⁻⁴² | Separated |
| Fully solved | Claude Opus 5.5 over DeepSeek V4.1 Flash | 7.6 × 10⁻³⁸ | Separated |
| Fully solved | Claude Opus 5.5 over GLM 5.3 Flash | 2.1 × 10⁻²⁶ | Separated |
| Fully solved | Claude Opus 5.5 over GPT-6 Luna | 2.1 × 10⁻³⁵ | Separated |
| Fully solved | Claude Sonnet 5.5 over Gemini 3.8 Flash | 1.1 × 10⁻⁴ | Separated |
| Fully solved | Claude Sonnet 5.5 over GPT-6 Sol | 3.3 × 10⁻²⁴ | Separated |
| Fully solved | Claude Sonnet 5.5 over DeepSeek V4.1 Flash | 2.8 × 10⁻¹⁸ | Separated |
| Fully solved | Claude Sonnet 5.5 over GLM 5.3 Flash | 2.3 × 10⁻¹¹ | Separated |
| Fully solved | Claude Sonnet 5.5 over GPT-6 Luna | 1.2 × 10⁻¹⁷ | Separated |
| Fully solved | Gemini 3.8 Flash over GPT-6 Sol | 1.2 × 10⁻¹⁴ | Separated |
| Fully solved | Gemini 3.8 Flash over DeepSeek V4.1 Flash | 2.1 × 10⁻⁷ | Separated |
| Fully solved | Gemini 3.8 Flash over GLM 5.3 Flash | 0.01 | Separated |
| Fully solved | Gemini 3.8 Flash over GPT-6 Luna | 1.6 × 10⁻⁷ | Separated |
| Fully solved | DeepSeek V4.1 Flash over GPT-6 Sol | 0.0014 | Separated |
| Fully solved | GLM 5.3 Flash over GPT-6 Sol | 8.3 × 10⁻⁹ | Separated |
| Fully solved | GPT-6 Luna over GPT-6 Sol | 8.8 × 10⁻⁵ | Separated |
| Fully solved | GLM 5.3 Flash over DeepSeek V4.1 Flash | 0.0015 | Separated |
| Fully solved | DeepSeek V4.1 Flash and GPT-6 Luna | 0.94 | Tie |
| Fully solved | GLM 5.3 Flash over GPT-6 Luna | 0.0028 | Separated |
| Fully solved in all 3 runs | Claude Opus 5.5 over Claude Sonnet 5.5 | 5.1 × 10⁻⁶ | Separated |
| Fully solved in all 3 runs | Claude Opus 5.5 over Gemini 3.8 Flash | 4.5 × 10⁻⁹ | Separated |
| Fully solved in all 3 runs | Claude Opus 5.5 over GPT-6 Sol | 3.7 × 10⁻²² | Separated |
| Fully solved in all 3 runs | Claude Opus 5.5 over DeepSeek V4.1 Flash | 1.3 × 10⁻²³ | Separated |
| Fully solved in all 3 runs | Claude Opus 5.5 over GLM 5.3 Flash | 2.3 × 10⁻¹⁷ | Separated |
| Fully solved in all 3 runs | Claude Opus 5.5 over GPT-6 Luna | 5.2 × 10⁻¹⁸ | Separated |
| Fully solved in all 3 runs | Claude Sonnet 5.5 and Gemini 3.8 Flash | 0.072 | Tie |
| Fully solved in all 3 runs | Claude Sonnet 5.5 over GPT-6 Sol | 4.4 × 10⁻¹² | Separated |
| Fully solved in all 3 runs | Claude Sonnet 5.5 over DeepSeek V4.1 Flash | 5.1 × 10⁻¹² | Separated |
| Fully solved in all 3 runs | Claude Sonnet 5.5 over GLM 5.3 Flash | 1.7 × 10⁻⁸ | Separated |
| Fully solved in all 3 runs | Claude Sonnet 5.5 over GPT-6 Luna | 1.0 × 10⁻⁷ | Separated |
| Fully solved in all 3 runs | Gemini 3.8 Flash over GPT-6 Sol | 4.1 × 10⁻⁹ | Separated |
| Fully solved in all 3 runs | Gemini 3.8 Flash over DeepSeek V4.1 Flash | 2.4 × 10⁻⁶ | Separated |
| Fully solved in all 3 runs | Gemini 3.8 Flash over GLM 5.3 Flash | 2.9 × 10⁻⁴ | Separated |
| Fully solved in all 3 runs | Gemini 3.8 Flash over GPT-6 Luna | 1.9 × 10⁻⁴ | Separated |
| Fully solved in all 3 runs | GPT-6 Sol and DeepSeek V4.1 Flash | 0.61 | Tie |
| Fully solved in all 3 runs | GPT-6 Sol and GLM 5.3 Flash | 0.078 | Tie |
| Fully solved in all 3 runs | GPT-6 Luna over GPT-6 Sol | 0.022 | Separated |
| Fully solved in all 3 runs | DeepSeek V4.1 Flash and GLM 5.3 Flash | 0.31 | Tie |
| Fully solved in all 3 runs | DeepSeek V4.1 Flash and GPT-6 Luna | 0.26 | Tie |
| Fully solved in all 3 runs | GLM 5.3 Flash and GPT-6 Luna | 1.0 | Tie |
The key-check signals
Of 153 approved keys, 19 carried a signal and 18 were re-read, capped at eight per country. Each signal is its own column, with no combined score:
- Reference self-check. Three judges grade the key’s own reference; a criterion two of three say it misses marks the key inconsistent.
- Recompute. The item is high or medium in the recompute triage.
- Law currency. The sweep of dated law changes proposed a fix.
- Expert fail, judges met. A specialist failed a criterion all three judges read as stated.
- Low-fail approver. The approver fails under 10% of criteria; this only orders the queue.
The self-check was rare: 8 of 853 criteria failed their own reference under two judges, 7 under all three. It tests consistency, not correctness.
