EmBench Global Work v1.0 technical report

Building the questions

The questions are written to match what our teams answer every day: notice periods, payroll cut-offs, statutory benefits, holiday calendars, termination rules and eligibility. Real questions from our users and internal teams.

Every question can be answered from Deel’s internal knowledge base: about 43,500 pages, around 69 million tokens, close to double its size a year ago. And it moves: since August 2026, about 15% of pages change every month. It is the same knowledge base the frontier models search and Pangaea K1 studied. The questions themselves are held out of Pangaea K1’s training data.

For each question, a model drafts a reference answer and breaks it into individual facts. Each fact is marked necessary (the answer is wrong without it) or supporting (useful, but optional). The 601 questions carry 2,788 facts: 4.6 per question on average, from 1 to 10, with a median of 4.

Drafting with a model keeps the set fast to build and cheap to refresh. It also leaves a gap a model is poorly placed to fill. Every fact describes something a correct answer should say, and none describes something it must not say, so the benchmark does not yet test whether a model states a plausible falsehood. Writing a convincing one is hard for a model and easy for a specialist.

Who checks the facts?

Every score in this report rests on two things: the judge that reads each answer, and the reference facts it checks that answer against. Reading the judge’s decisions tells us when the judge is wrong. It cannot tell us when a reference fact is wrong, missing or out of date. The judge will score every answer against that fact, consistently, and the error will look like signal.

So, through Emet, Deel specialists who answer these questions every day are checking the benchmark itself. They verify each fact against the law and Deel practice, with a source, and correct it where it is wrong. They add the facts that are missing, and false statements that a correct answer must reject. On a sample of answers they also make the same calls as the judge, so we can see how far to trust it.

In an earlier round, in which specialists graded Pangaea’s answers directly, we learned that specialist time is best spent explaining their thinking: what was wrong and, often, what the right answer is.

Facts that move, facts that stay

Not every fact ages at the same speed. A holiday calendar is published once and then stays put; a fee schedule may be edited every few weeks. So we tag every fact as stable or volatile, based on what the page’s edit history shows rather than on what the fact looks like. For each page the benchmark draws on, we collect the visible-text edits from the last 90 days. A judge model then checks, fact by fact, whether any edit changed that fact’s value: a rate, a date, a provider, a yes or no. Rewording, moving or reformatting a fact does not count. Pages that keep revising facts faster than our retraining cycle are flagged as a whole.

We score volatile facts separately, so a stale answer is never mistaken for a learning gap.

Scoring facts, not impressions

A judge model reads each answer and checks each fact in turn: is it there, and is it stated correctly? A question’s score is the share of its facts the answer gets right, and a model’s score is that share across all the questions it answered. A question counts as all pass when every one of its facts is right.

This takes more work than asking a judge to grade an answer out of ten, but it pays off. A low score points at the exact facts that were missed, which tells us where to look.

We also keep an eye on what the score rewards. A reference answer can include facts the question never asked for, and a long answer is more likely to hit them by accident. So we report a second score that counts only necessary facts, the ones an answer is wrong without. On that view Claude Opus 5.5 leads at 79, and Gemini 3.8 Flash rises to 78.

Checking the checker

The judge is a model too, and it can be wrong. When we compared judge models on the same answers, they sometimes ranked candidates very differently, and one was rewarding answers that sounded plausible without checking the facts. We replaced it, and we now read samples of the judge’s decisions, not only its averages.

Studies on general chat questions find that strong model judges agree with people about as often as people agree with each other. Through Emet, we are measuring whether that holds for employment rules across more than 150 countries.

The headline numbers

Every system answered the same 601 questions through the same harness and the same judge. The eight frontier models had search tools over our knowledge base that they could call while answering, as they would in production. Pangaea K1 answered from memory, with no tools. The times we quote are medians for the complete answer in our evaluation harness; time to the first word in production is a different, much smaller number.

We ran Claude Opus 5.5, Sonnet 5.5 and Sonnet 5, GPT-6 Sol and Luna, Gemini 3.8 Flash, DeepSeek V4.1 Flash and GLM 5.3 Flash. Their scores ranged from 52 to 75. Pangaea K1, answering from memory alone, scored 70. That puts it ahead of Gemini 3.8 Flash, Claude Sonnet 5 and both GPT-6 models, five points behind the best, and faster than all of them.

All pass is well below the fact-level score for every system. DeepSeek V4.1 Flash states 75% of facts but fully solves 64.5% of questions. No system fully solved 137 of the 601 questions (23%).

These numbers do not compare directly with EmBench Payroll. Payroll goes deep on payroll in 8 countries, scores models closed book on criteria drafted by a model and corrected by specialists, and runs each model three times. Here, frontier models search our knowledge base and Pangaea K1 answers from memory. The settings and metrics differ, so the two boards rank models differently.

Retrieval still has a clear place. The best models with search score a few points above Pangaea K1, and search is the only reliable source for facts that change faster than we retrain, such as rates and fee schedules. But an answer with search is only as good as the pages the search returns, and it is slow. A model with search tools searches, reads, often searches again, and only then writes. The models with search took a median of 13.1 to 28.4 seconds per answer. Pangaea K1, answering from memory, took 7.7. In the product, where answers stream, its first words appear within a second of asking.

The EmBench Global Work page plots score against median seconds per answer for every system, including Pangaea K1 and the base 9B model before training.

The base 9B model makes the same point from the other side. Before any training on Deel’s knowledge it scored 19 from memory, and 45 when given the same search tools. Continued pretraining on our knowledge base took it to about 54, and fine-tuning to 70, without searching at all.

An earlier run of the same benchmark compared Pangaea question by question with one frontier model using search, on the 561 questions both answered. They tied on about half the questions (52%), and Pangaea won outright on about one in five (20%), against 29% for the model with search. Pangaea answered 48% of questions perfectly, against 52%, and scored zero on 17%, against 14%.

The EmBench Global Work page shows this question-by-question comparison.

Where it currently falls short

The clearest weakness is questions that need facts from two or more pages. Pangaea K1 scores about 57 on those, some 15 points below its score on questions a single page answers: it struggles to connect facts it learnt separately in training.

One thing that is not the problem is freshness. Of the 755 facts Pangaea K1 missed, 752 were stable: their pages had not revised that fact recently. The gaps come from what the model learned, not from the knowledge base moving on.

Both are part of what comes next. A light retrieval step will confirm specifics when the model is unsure, and the incremental training pipeline will refresh only the pages that change, with retrieval covering the volatile facts that change faster than we retrain.

From a score to a fix

Fact-level scores make problems traceable. One issue they showed is that questions about pages that share text with other pages scored noticeably lower. We traced this back to our training corpus: shared text had been kept for only one page, so the other pages were missing part of their content. The next training corpus will fix this.

We found most real problems by reading raw output, before any metric showed them. One early training corpus gave long chunks, which held 81% of the text, almost no restatements, so the model saw each fact only four or five times. We spotted it by reading samples.

We read these results as validation of the approach, not as a finish line. Answering from memory gets most of the way there, quickly. Retrieval still adds the last few points, which is why our goal is less search, not no search.

What we would tell a team building their own

• Measure facts, not impressions. Fact-level scoring turns a vague “it seems worse” into a specific fix.
• Check the checker. Read samples of the judge’s decisions, and have specialists check the reference facts it scores against.
• Tag what moves. Score volatile facts separately so a stale answer is not mistaken for a gap in learning.
• Read the samples. Problems show up in raw output long before they show up in a dashboard.

Room to improve

No system fully solved 137 of the 601 questions (23%), with search or without, so there is room for every system to improve, Pangaea K1 included. This is one of the advantages of a training set which sets the ceiling above just the way the system is currently used, and we will keep publishing improvements as we continue development.

Dataset card

Further reading

• Allen-Zhu & Li, Physics of Language Models, Part 3.1: Knowledge Storage and Extraction (2023)
• Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2023)
• Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (2024)
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
• Harvey, Harvey Tenet research preview (see the Firm Knowledge section)
• Deel, Pangaea series, Parts 1–4