TL;DR
Read the benchmark name, scoring rule and evaluation conditions together. GPQA, AIME and SWE-bench test different abilities, so their percentages cannot be treated as a single measure of intelligence. Check the dataset version, tools, reasoning budget and attempt count before comparing models. Missing results are unknown, not zero; pass@k and success on every repeated attempt describe different outcomes.
01 / Start with the names in a model launch table
GPQA Diamond measures difficult science question answering. AIME measures competition mathematics. SWE-bench measures whether an agent can fix software issues. These are different tests, so their percentages describe different abilities. This guide explains the benchmark names first, then the scoring labels and evaluation settings printed beside them.
The examples below come from representative official DeepSeek, OpenAI, and Claude releases, including older releases that established familiar labels. They are references for reading a launch table, not a current model ranking. Numerical examples elsewhere in this guide are illustrative unless explicitly attributed.
DeepSeek-R1: official evaluation tables ↗DeepSeek-V4-Pro: official model card ↗OpenAI: introducing GPT-5 ↗OpenAI: introducing GPT-5.4 ↗Anthropic: Claude Opus 4.6 evaluation chart and notes ↗
| Official release examples | Labels you will encounter | What to look up below |
|---|---|---|
| DeepSeek-R1 and DeepSeek-V4-Pro | MMLU-Pro (EM), GPQA Diamond (Pass@1), Codeforces (Rating), SWE Verified (Resolved) | What the test measures, followed by the meaning of the metric in parentheses. |
| OpenAI GPT-5 and GPT-5.4 | AIME, GPQA Diamond, SWE-bench, MMMU, GDPval, MRCR | Mathematics, science, coding, vision, work products, and long-context retrieval. |
| Claude Opus 4.6 | Terminal-Bench, OSWorld, τ²-bench, MCP Atlas, BrowseComp, HLE, MMMU-Pro, MMMLU | Terminal work, computer operation, tool use, research, and academic reasoning. |
02 / MMLU, MMLU-Pro, MMMLU, and Chinese knowledge tests
These benchmarks ask exam-style questions across many subjects. They are useful for understanding knowledge breadth and question-answering performance. A high overall score can still hide weak subjects or languages.
Watch the spelling: MMMLU is multilingual text evaluation; MMMU, covered under vision below, includes images. They are different benchmarks.
MMLU: original benchmark ↗MMLU-Redux: dataset corrections ↗MMLU-Pro: tasks and answer options ↗OpenAI: MMMLU dataset ↗C-Eval: benchmark paper ↗CMMLU: Chinese subject evaluation ↗
| Benchmark | What the model does | What the score means |
|---|---|---|
| MMLU | Answers multiple-choice questions across subjects such as history, law, mathematics, and medicine. The name expands to Massive Multitask Language Understanding. | Accuracy / EM / pass@1: how often the extracted answer is correct under the reported protocol. |
| MMLU-Redux | Answers a revised MMLU question set with annotations addressing problems in the original data. | A score on the revised set; check the Redux version before comparing it with another result. |
| MMLU-Pro | Answers more reasoning-focused questions with up to ten choices, rather than the original MMLU's four. | Correct-answer rate on Pro. An 85 on Pro and an 85 on original MMLU describe different question sets. |
| MMMLU | Answers professionally translated MMLU questions in multiple languages. | Check whether the number is a particular language or an aggregate, and which languages contribute. |
| C-Eval / CMMLU | Answers Chinese exam and subject-knowledge questions. These are separate datasets; CMMLU also includes knowledge specific to Chinese contexts. | Usually answer accuracy or EM. Neither number is a general score for the quality of Chinese writing. |
03 / GPQA Diamond and HLE — difficult academic questions
These rows are commonly used to show scientific and academic reasoning. A correct final answer earns credit under the benchmark's grader; an impressive explanation alone does not establish correctness.
GPQA: questions and subsets ↗Humanity's Last Exam: dataset and evaluation ↗
| Benchmark / qualifier | What it tests | How to read the result |
|---|---|---|
| GPQA Diamond | Graduate-Level Google-Proof Q&A: multiple-choice biology, chemistry, and physics questions. Diamond is a 198-question subset selected using expert and non-expert validation. | Accuracy or pass@1 measures accepted choices. Diamond is a subset label, not a reasoning mode or a percentage threshold. |
| GPQA Main | The broader 448-question GPQA set. | Main and Diamond have different questions and denominators. Compare the same subset. |
| HLE / Humanity's Last Exam | Difficult, closed-ended questions across academic subjects, including image-dependent questions in the full dataset. | Accuracy / pass@1 measures accepted answers. A text-only result excludes part of the multimodal test. |
| HLE with tools / without tools | Whether an attempt can use capabilities such as web search and code execution. | With-tools results measure the model together with its available tools. Check the text/image subset as well as the tool setting. |
04 / GSM8K, MATH-500, AIME, HMMT, and FrontierMath
Math scores usually grade the final answer. School word problems, competition problems, and research-level mathematics impose very different demands. The exam year, question subset, and permission to execute code are part of the result.
GSM8K: dataset paper ↗MATH-500: dataset card ↗MAA: AIME format ↗HMMT: rounds and scoring ↗Epoch AI: FrontierMath tasks and tiers ↗
| Benchmark | What the model solves | Score and important qualifiers |
|---|---|---|
| GSM8K | Grade-school word problems requiring several arithmetic steps, such as combining quantities and calculating a remainder. | Usually final-answer accuracy. GSM8K's difficulty is very different from olympiad or research mathematics. |
| MATH-500 | A 500-problem subset of MATH, covering competition-style algebra, geometry, number theory, and other topics. | Usually pass@1 or answer accuracy. The 500 specifies the subset size; it is not a maximum score. |
| AIME 2024 / 2025 / another year | American Invitational Mathematics Examination questions. Each paper has 15 problems with integer answers from 000 to 999. | Usually pass@1. The year and whether AIME I, II, or both are included matter. On one 15-question paper, 12 correct is 80%. |
| HMMT | Harvard–MIT Mathematics Tournament problems. February individual rounds include algebra and number theory, geometry, and combinatorics. | Check the month, year, and included rounds. Unweighted AI answer accuracy is different from the contest's weighted points. |
| FrontierMath | Expert-written advanced mathematics problems, including research-level problems that can require substantial computation. | Solved-problem rate under the specified tools and budget. Tiers 1–3 and Tier 4 are different difficulty groups; keep the tier and dataset version attached. |
05 / HumanEval, LiveCodeBench, Codeforces, and Aider
These coding rows test writing or editing programs. Passing programming tests, ranking in a contest, and applying edits through a coding agent are distinct outcomes. In particular, Codeforces Rating is not a percentage.
HumanEval: functional correctness ↗LiveCodeBench: scenarios and releases ↗DeepSeek-R1: Codeforces rating and percentile columns ↗Aider: polyglot benchmark design ↗Aider: run details and pass rates ↗
| Benchmark / metric | What the model does | What the number means |
|---|---|---|
| HumanEval — pass@1 | Completes a small programming function from its signature and description; the evaluator executes tests. | Estimated probability that one sampled implementation passes the tests. It does not measure completion of an entire software project. |
| LiveCodeBench — pass@1 | Solves programming problems collected over time. It also has other scenarios, including self-repair and code execution. | For code generation, the share of successful sampled programs. Keep the release, problem-date window, and scenario attached to the score. |
| Codeforces — Rating | Solves competitive programming tasks under a contest evaluation protocol. | A rating on the contest scale. A value such as 2,000 is a rating, not 2,000 correct answers or 2,000%. Check how the evaluator estimates the rating. |
| Codeforces — Percentile | Places the evaluated performance relative to the stated comparison population. | The 96th percentile is approximately above 96% of that population. It does not mean 96% of problems were solved. |
| Aider-Polyglot — accuracy | Edits exercise files across six programming languages through Aider, then runs tests. The standard benchmark permits a second attempt after test feedback. | Share of exercises solved under that editing protocol. Check first-attempt versus final pass rate, edit format, and Aider version. |
06 / SWE-bench and Terminal-Bench — software agents at work
An agent can inspect files, make edits, run commands, and react to results. One attempt may contain many model responses and tool calls. These benchmarks therefore measure a model operating inside a particular agent, with a particular budget.
SWE-bench Verified: human-reviewed tasks ↗SWE-bench: datasets and evaluation ↗SWE-bench: repair and regression grading ↗SWE-bench Pro: versions and splits ↗Terminal-Bench: version 2.0 methodology ↗
| Benchmark | Task and success condition | What to check beside the score |
|---|---|---|
| SWE-bench Verified | Given a repository and an issue, produce a patch that passes the required repair and regression tests. Verified is a human-reviewed set of 500 issues. | Resolved (%) is the proportion of issues accepted. On all 500 tasks, 400 resolved means 80%. It does not mean 80% of every patch is correct. |
| SWE-bench Pro | Resolve repository tasks from a separate benchmark with its own repositories, environments, and grading protocol. | Pro is not a paid tier of Verified. Check dataset version, public or private split, and whether a smaller hard subset was used. |
| SWE-bench Multilingual | Resolve repository issues across multiple programming languages. | Multilingual refers to programming languages here. Use the same task collection and agent setup when comparing resolved rates. |
| Terminal-Bench | Complete tasks inside a terminal, such as building software, processing data, or configuring an environment; a verifier checks the outcome. | Accuracy / success rate is task completion, not command accuracy. Version, agent, timeout, and how repeated trials are aggregated all matter. |
07 / τ-bench, τ²-bench, MCP Atlas, and Toolathlon
Tool-use tests ask whether an agent can complete a task using external functions and services. Choosing a function is only one step: the agent may also need to follow policy, ask for information, pass correct arguments, and verify the final state.
τ-bench: task success and pass^k ↗τ²-bench: domains and coordinated interaction ↗MCP Atlas: benchmark paper ↗OpenAI GPT-5.4: Toolathlon examples and tool-use conditions ↗
| Benchmark | What the model does | How the score is interpreted |
|---|---|---|
| τ-bench / tau-bench | Handles simulated customer-service conversations using domain tools and policy rules, such as making an eligible order change. | Task success includes the required outcome, such as the correct database state. pass^k measures reliability across k trials, with every trial required to succeed. |
| τ²-bench / tau2-bench | Extends the interaction setting so the simulated user can also act in the environment; the agent and user may need to coordinate. | Retail, airline, and telecom are domain labels. A telecom result is not automatically an overall score. The superscript 2 in the name is the benchmark generation, not two attempts. |
| MCP Atlas | Finds and uses appropriate tools exposed through Model Context Protocol servers, including selecting useful tools among distractors. | Pass rate summarizes completed tasks. Public versus private tasks, tool discovery, and the available tool set affect the result. It is not a count of supported MCP servers. |
| Toolathlon | Executes multi-step workflows across software services—for example, reading submitted files, calculating results, and recording them in a spreadsheet. | Success concerns completing the workflow under its evaluator. Tool access, environment, and version must accompany the percentage. |
08 / OSWorld and ScreenSpot-Pro — operating a computer
Computer-use evaluations cover both recognizing where to act and completing a sequence of actions. Locating the right button is a narrower task than successfully editing a document or finishing a workflow.
OSWorld: tasks and execution-based evaluation ↗ScreenSpot-Pro: target grounding methodology ↗OpenAI GPT-6 Astra: versioned computer-use scoring notes ↗
| Benchmark | What it tests | Score and qualifiers |
|---|---|---|
| OSWorld | Completes tasks in real desktop and web applications, potentially across multiple apps. The evaluator checks the resulting state. | Read the version's completion or partial-credit rules. Step limits, screenshot handling, and the computer-use agent affect performance; a newer version is a different evaluation. |
| ScreenSpot-Pro | Given an instruction and a high-resolution application screenshot, locates the requested interface element. | Grounding accuracy measures whether the predicted point lands in the target region. It does not establish that a full multi-step workflow can be completed. |
09 / MMMU, MMMU-Pro, and OmniDocBench — understanding images
These rows evaluate visual inputs: diagrams, charts, document pages, and other images. The model must extract relevant information from the image and, where required, reason with it.
MMMU: multimodal tasks ↗MMMU-Pro: evaluation variants ↗OmniDocBench: document parsing evaluation ↗OpenAI GPT-5.4: normalized edit distance reporting ↗
| Benchmark | What the model does | How to read the score |
|---|---|---|
| MMMU | Answers college-level questions combining text with subject-specific images, such as scientific diagrams, charts, and artwork. The name includes Multimodal Understanding. | Answer accuracy on the specified split. This is image-and-text reasoning, unlike the multilingual text benchmark MMMLU. |
| MMMU-Pro | Takes a more demanding version designed to reduce questions answerable without the image; it includes revised answer options and vision-focused settings. | Check the standard or vision setting and whether tools are permitted. Pro and original MMMU are separate test results. |
| OmniDocBench | Parses document pages into structured content, including text, tables, and formulas. | Read the named metric: normalized edit distance measures extraction error, so lower is better. Do not assume every number on a vision table is a higher-is-better percentage. |
10 / MRCR and LongBench v2 — using long context
A context window describes how much input a model can accept. A long-context benchmark tests what it can actually retrieve or reason about inside that input. The input length belongs beside the score.
OpenAI MRCR: needles, lengths, and scorer ↗LongBench v2: tasks and evaluation ↗
| Benchmark / field | What it means | How to interpret it |
|---|---|---|
| MRCR | Multi-Round Co-reference Resolution: retrieve a particular earlier response from a long conversation containing similar requests and distractors. | Measures whether the right earlier content is recovered, rather than merely whether the prompt fits in memory. |
| 2-needle / 4-needle / 8-needle | The number of competing target-like items in the MRCR conversation. | More needles make identifying the requested occurrence harder. This number is not the number of generated attempts. |
| MRCR match ratio / MMR; length bins | A retrieval similarity score under the stated grader. The public dataset uses a string similarity ratio after checking a required prefix. | A match score may award partial credit. A 512K–1M result concerns that length range; it does not describe performance at every context length. Check dataset version too. |
| LongBench v2 | Multiple-choice reasoning over long documents, conversation histories, code repositories, and structured data. | Usually answer accuracy / EM. Check context-length and difficulty groups. LongBench v2 is not the same dataset or scoring scheme as original LongBench. |
11 / IFEval, DROP, and ARC-AGI — instructions and reasoning
These benchmarks test different forms of correctness: obeying explicit constraints, extracting and calculating answers from a passage, or discovering a new visual rule. Their scores should not be treated as interchangeable reasoning percentages.
IFEval: verifiable instructions and scoring ↗DROP: reasoning and answer metrics ↗ARC Prize: grid tasks and benchmark versions ↗
| Benchmark | Concrete task | Meaning of the scoring label |
|---|---|---|
| IFEval / IF-Eval | Follow mechanically checkable instructions, such as including required words or producing a specified number of paragraphs. | Prompt Strict requires every instruction in a prompt to pass the strict checks. Instruction-level accuracy scores constraints individually; loose scoring allows defined normalization. |
| DROP | Read a passage and answer a question requiring operations such as counting, subtraction, or comparison. | EM requires a matching normalized answer. F1 measures answer-token overlap, with special numerical and multi-span handling. 3-shot F1 means three prompt demonstrations and F1 scoring. |
| ARC-AGI / ARC-AGI-2 | Infer a transformation from example pairs of colored grids, then apply it to a new input grid. | Exact output-grid success under the declared attempt and compute limits. Version 1 and 2 contain different tasks; a high score alone does not establish general intelligence. |
12 / SimpleQA and BrowseComp — factual answers and web research
Knowing a fact from the model's parameters and finding it using a browser are different capabilities. Read whether browsing is available before interpreting a factuality score.
SimpleQA: grading and abstentions ↗Chinese SimpleQA: dataset and evaluation ↗BrowseComp: final-answer grader ↗
| Benchmark | What the model does | What a score does—and does not—say |
|---|---|---|
| SimpleQA — Correct | Answers short factual questions with a verifiable reference answer. The original evaluation grades answers as correct, incorrect, or not attempted. | Correct rate is the fraction of all questions answered correctly. The remainder includes abstentions, so 40% correct does not necessarily mean 60% hallucinated. |
| Chinese SimpleQA / C-SimpleQA | Answers short factual questions in a separate Chinese benchmark. | Read its own question set and grader. It is not interchangeable with the English SimpleQA score. Likewise, a Verified variant identifies a changed dataset. |
| BrowseComp | Uses browsing to locate hard-to-find information and return a short, verifiable answer. | Accuracy / pass@1 concerns the accepted final answer from a research attempt. Search access, blocked sites, tool budget, and the evaluation date can change the task. |
13 / GDPval, Finance Agent, and preference benchmarks
Some evaluations judge completed work or compare responses rather than check a single exam answer. The task set, judge, and comparison baseline determine what these scores mean.
OpenAI: GDPval tasks and expert comparisons ↗Artificial Analysis: GDPval-AA methodology ↗Vals AI: Finance Agent v1.1 methodology ↗AlpacaEval: length-controlled win rates ↗
| Benchmark / metric | What is evaluated | How to read it |
|---|---|---|
| GDPval — wins / ties | Work products such as spreadsheets, presentations, and written documents for occupational tasks, compared with expert-produced reference work. | Win rate counts work judged better; wins-or-ties also includes comparable work. Neither directly measures the percentage of jobs an AI can replace. |
| GDPval-AA — Elo | Artificial Analysis' evaluation of GDPval tasks using its own agent setup and pairwise judging framework. | A relative rating within a particular version and comparison pool. A rating of 1,600 is not 160%, and GDPval-AA Elo is not OpenAI GDPval's win percentage. |
| Finance Agent | Researches company information and financial filings using tools to answer analyst-style questions. | Version 1.1 reports final-answer accuracy. Keep the version attached and check its rubric, tools, task split, and judge before comparing results. |
| AlpacaEval 2.0 — LC win rate | Compares instruction-following responses with a reference model using an automated judge. | LC means length-controlled: the estimate adjusts for response-length effects. A 60% win rate describes preference against that baseline, not 60% factual correctness. |
14 / What Accuracy, EM, F1, Resolved, and Elo mean
The benchmark names the task; the metric names the scoring rule. Most answer-accuracy and completion scores are higher-is-better. Error metrics such as edit distance are lower-is-better. A dash, N/A, or null indicates an unavailable result, not zero.
DROP: exact match and answer F1 ↗SWE-bench: resolution criteria ↗GDPval-AA: relative ratings ↗
| Label | Definition | Illustrative reading |
|---|---|---|
| Accuracy / Acc. / Correct (%) | Correctly graded answers divided by evaluated answers, multiplied by 100. | 150 correct out of 200 = 75%. Always identify which questions were included. |
| EM / exact match | The extracted answer must match a reference after the evaluator's defined normalization. | No partial credit for a merely similar answer. Normalization may ignore case, punctuation, or equivalent formatting, depending on the grader. |
| F1 | The harmonic mean of precision and recall. In answer-overlap grading, precision measures how much of the predicted answer is relevant; recall measures how much of the reference it covers. | With precision 1/2 and recall 1, F1 = 2PR / (P + R) ≈ 0.667, often displayed as 66.7. Benchmark-specific normalization still applies. |
| Resolved / success rate | Fraction of tasks meeting the evaluator's required completion criteria. | 80 accepted patches out of 100 issues = 80% resolved. Partial-credit task scores must be labeled separately. |
| Elo / rating | A relative performance scale fitted or computed under the benchmark's ranking method. | Higher is better within the same system. Different benchmarks' ratings have different baselines and cannot be compared directly. |
| Percentage points / relative improvement | An absolute difference between percentages versus that difference divided by the starting value. | 80% to 84% is +4 percentage points, or a 5% relative increase. |
15 / pass@1, pass@k, pass^k, and cons@64
These labels answer different questions about repeated attempts. pass@1 can be estimated from many generated samples: DeepSeek-R1's official evaluation notes describe 64 responses per question for benchmarks requiring sampling, while reporting pass@1. That does not make the score pass@64.
For the standard coding estimator, n is the number of generated candidates and c is the number that pass. For each problem, pass@k = 1 − C(n − c, k) / C(n, k), with n ≥ k; C counts combinations. Average the per-problem estimates to obtain the dataset score.
HumanEval: pass@k estimator ↗τ-bench: repeated-success reliability ↗DeepSeek-R1: sampling and cons@64 ↗Self-consistency: answer voting ↗
| Label | Question being scored | Illustrative example |
|---|---|---|
| pass@1 | Would one sampled attempt succeed? Repeated samples can estimate this probability. | For one task, 2 passing candidates out of n = 10 estimate pass@1 at 20%. |
| pass@k | Would at least one of k candidates succeed? It assumes a checker can identify success. | With n = 10 and c = 2, estimated pass@5 = 1 − C(8,5)/C(10,5) ≈ 77.8%. |
| pass^k | Would all k repeated attempts succeed? This measures consistency. | Assuming independent attempts each with success probability 80%, pass^3 is 0.8³ = 51.2%, while pass@3 is 99.2%. |
| cons@64 / maj@k | Generate multiple answers and grade the answer selected by majority voting under the stated normalization and tie rules. | If A is correct but the five samples are A, A, B, B, B, the majority vote fails even though pass@5 succeeds. |
| avg@k / repeated-run mean | Usually the mean score across k runs; confirm the paper's definition. | Run accuracies of 60% and 80% average to 70%. This is different from counting any success across both runs. |
| Best-of-k | Generate k candidates, then let a stated ranker or verifier select one for final grading. | A correct candidate may exist but not be selected, so selected-answer accuracy need not equal pass@k. |
16 / Reading the small print: shots, tools, budgets, and versions
A complete result identifies the model, benchmark version or subset, metric, and evaluation conditions. For example, “DROP (3-shot F1)” means passage reasoning scored with answer F1 after three prompt demonstrations. “SWE-bench Verified (% resolved)” means accepted issue fixes under the stated agent setup. “MRCR v2, 8-needle, 512K–1M” identifies a dataset version, distractor setting, and context-length range.
DeepSeek-R1: generation settings and reporting ↗Claude Opus 4.6: evaluation conditions and trial counts ↗OpenAI GPT-5.4: tools, subsets, and length bins ↗Transformers: temperature, top_p, and generation limits ↗
| Footnote or parameter | Meaning | Why it changes the reading |
|---|---|---|
| 0-shot / 3-shot / 5-shot | Number of demonstrations provided before the test question. | 5-shot means five prompt examples, not five attempts at the question. |
| CoT / thinking / reasoning effort | CoT requests intermediate reasoning; thinking and effort settings control model-specific reasoning behavior or resources. | High, max, and xhigh are provider-specific settings, not standardized amounts of computation across models. |
| With tools / no tools / TIR | Whether the model may call external tools; TIR means tool-integrated reasoning, often including code execution. | Search, Python, browsing, and other tools can change what the system is able to solve. |
| Token budget / temperature / top_p | Maximum allowed generation and sampling controls. Temperature changes how concentrated token selection is; top_p limits sampling to a cumulative-probability set. | These are run conditions. A larger budget allows more work but does not guarantee a better answer; matching one setting does not equalize all compute. |
| Agent / scaffold / harness; time or step limit | Software that manages the model's tools and actions, the evaluation runner, and the resources allowed for a task. | Two agents using the same model may produce different results. A 100-step allowance is not 100 independent solutions. |
| Year / version / subset / asterisk | The actual questions and special conditions behind the number. | AIME 2024 versus 2025, Verified versus Pro, and HLE text-only versus full multimodal require separate labels. Read the specific note attached to an asterisk. |
| Mean / ± / confidence interval | How repeated measurements are aggregated and how uncertainty is summarized. | ± may denote a standard deviation, standard error, or confidence interval; read its label. Do not automatically treat every ± value as a 95% interval. |
