Humanity's Last Exam: What the Score Means

Humanity's Last Exam (HLE) is a 2,500-question test of expert-level academic knowledge, built by the Center for AI Safety (CAIS) and Scale AI. Every question has a single checkable answer, and the questions were chosen because frontier models failed them. The lastexam.ai page we read on 31 October 2026 shows scores from 2.7% to 38.3% on its table, and vendors now report much higher figures when tools are allowed.

The short version

What it measures

The paper's abstract gives the motive. Models were passing 90% on popular benchmarks like MMLU, which limits informed measurement of the best systems. HLE was built as a multi-modal benchmark at the frontier of human knowledge, "designed to be the final closed-ended academic benchmark of its kind with broad subject coverage." It has 2,500 questions across over a hundred subjects. The authors release the questions publicly and keep a private test set of held-out questions to check for overfitting.

The lastexam.ai page adds how the questions were gathered. They came from nearly 1,000 subject expert contributors affiliated with over 500 institutions across 50 countries. A question was kept when frontier models could not answer it, so the set is hard by construction. Questions are closed-ended: a short exact answer or a multiple-choice option, which lets the grader mark them automatically. Some questions include images, which is why the page notes that a model that is not multi-modal is evaluated on a text-only subset.

That design has a clear strength and a clear limit. It tests deep recall and reasoning in specialist fields, at a difficulty ordinary tests no longer reach. It does not test open-ended judgement, long projects or taste.

Accuracy and calibration

HLE reports two numbers. Accuracy is the share of questions answered correctly. Calibration error measures how far a model's stated confidence is from its real hit rate. The page explains that models are prompted to give an answer and a confidence from 0% to 100%. A model that says "90% sure" and is right half the time is miscalibrated. Lower calibration error is better.

Calibration matters for real use. A tool that is often wrong but knows when it might be wrong is easier to use safely than one that is wrong and confident. The table on lastexam.ai shows the effect: the best model on accuracy is not the best on calibration.

Published figures from lastexam.ai

This is the table on the lastexam.ai home page, read on 31 October 2026. The page labels it with judge model o3-mini and "Dataset Updated: April 3rd, 2025." It is the project's own table, and we did not run any of it.

Notes. The asterisk on DeepSeek-R1 on the page marks a text-only evaluation. The page also points to the CAIS AI Dashboard and the SEAL LLM Leaderboards, which show the same data. Those models are from 2025. The models in our model pages, released in 2026, are not on this table.

Vendor-reported figures for current models

Model makers report HLE in their launch posts, usually with and without tools. These are vendor-reported and copied from our model pages, which read Anthropic's posts on 31 October 2026.

Compare these with 38.3% on the project table and the gap is large. It comes from at least three things: newer models, tools allowed (search and code execution change what a model can do), and a different harness and grader. The project's table is labelled as a particular dataset version and judge; the vendor figures follow each vendor's own settings. Read the footnotes on Claude Opus and Claude Sonnet before comparing across rows.

With tools versus without tools

This is the most common misreading. Without tools, a model answers from its trained knowledge. With tools it may search the web, run code and read files. A search tool can look up an exact fact. A question that was built to defeat memory may fall to a search. So "HLE 67.7% with tools" and "HLE 45% without tools" describe two different capabilities, and a table that mixes them is wrong. A fair statement names which one, and says what tools were allowed.

The newer variants

The lastexam.ai news list tells you how the project has moved:

• 3 April 2025. HLE was finalised at 2,500 questions.
• 8 October 2025. A dynamic fork, HLE-Rolling, was released with an update log, so the question set can change over time.
• 28 January 2026. The paper appeared in Nature.
• 22 September 2026. HLE-Diamond was released. The page says its goal is to provide a seamless migration path for researchers once frontier AI begins to hit the noise ceiling on the original HLE dataset.

The last item is the one to notice. The authors themselves expect the original set to become noisy as top scores rise, partly because some questions and answers in any large expert-written set will be wrong or ambiguous. When a vendor writes only "HLE," ask which version they used.

Worked example: what calibration error looks like

This is an example with made-up numbers, not a published result. Suppose a model answers 100 questions and, for each, states a confidence.

Total correct is 40, so accuracy is 40%. The model's average stated confidence was well above that, so its calibration error is large. A user who trusts the 90% answers will be wrong half the time. Calibration error summarises that gap as a single percentage. On the lastexam.ai table every listed model shows calibration error of 50% or more, which tells you that, at least on these questions, stated confidence was a poor guide. A careful user treats a confident answer to a hard expert question as a lead to check, not as a fact.

Why the gap between 38.3 and 67.7 is not a contradiction

A reader who sees 38.3% on the project's table and 67.7% in a launch post may think one is wrong. Both can be true, because the cells differ on several things at once.

Because five things move together, you cannot read the difference as a measure of progress alone. If the 2026 vendor figure came with a no-tools row, the comparison would be fairer. Haiku's launch post, for example, shows both with-tools and no-tools rows, and the Anthropic pages on this site note that Opus 5.5 results use adaptive thinking at maximum effort unless noted.

How HLE differs from older knowledge tests

None of this makes HLE a measure of intelligence. It makes it a harder and better-guarded measure of one thing, expert recall and reasoning on closed questions.

HLE and agents

HLE is a model test, not an agent test, but the with-tools row blurs the line. A model with search and code execution is acting like a small agent. If your use is research help, the with-tools number is the relevant one, and it depends on the tools you provide. Search servers such as Exa, Tavily and Brave Search are the sort of add-on that changes a result like this, and their quality varies. For agents that edit code, other tests fit better. For work products, see GDPval.

Limits and caveats

Closed-ended

Every question has one right answer. Real research and real work rarely do. A model can be excellent at HLE and poor at writing a good proposal.

Label noise

With 2,500 expert-written questions, some will have wrong or disputed keys. The authors' own plan for a Diamond subset suggests they see this as a growing issue near the top of the scale.

Grader dependence

Answers are checked by a judge model. The project's table names o3-mini as its judge. Different judges and prompts can shift scores by points. A vendor that uses its own grader is reporting something close to, but not exactly, the project's number.

Tool access changes everything

See above. It is the first thing to check.

Public questions

The questions are public on Hugging Face as cais/hle. A private held-out set exists to detect overfitting, but a model trained on the public set may look better than it is.

How to read an HLE claim

1. Which version: original, Rolling or Diamond?
2. With tools or without, and which tools?
3. Who graded, and with which judge model?
4. Is the number vendor-reported or on the project's own board?
5. Is the calibration error shown, or only accuracy?
6. What is the date?

What to do with HLE

A short home test

Take ten questions you know the answers to in your own field, each with one unambiguous answer. Ask two models, once with search turned off and once with it on. Count correct answers and ask each model for a confidence figure. You will see two things quickly: how much tools change the result, and whether the model's confidence means anything. That is HLE's method at kitchen-table scale, and it tells you more about your use than the public number.

Checklist before you quote an HLE number

• You say which version of the dataset it is.
• You say "with tools" or "without tools", and list the tools if you know them.
• You name the source: the project board or a vendor post, with its date.
• You do not set a vendor figure beside a project figure as if they were one series.
• You mention calibration if you mention trust.
• You do not use it to rank models for work it does not test.

Our take

HLE is a useful stress test for knowledge and reasoning, with a thoughtful design: expert questions, a held-out set, a calibration measure and honest talk about saturation. Its headline number is also the most easily abused, because tools and graders move it a lot. We cite the project's table with its date, label vendor figures as vendor-reported, and do not use HLE to rank coding agents. We have not run it ourselves. See the benchmark hub, Claude models and our methodology.