A Better LLM Grader for Humanity’s Last Exam

glVersion = "dev"
glr = import(`/assets/gl-react.js?v=${glVersion}`)
hle = import(`/blog/posts/hle-judge-comparison/components.js?v=${glVersion}`)

This post outlines our approach to updating the grader for Humanity’s Last Exam. We hope that sharing this detail will provide transparency into our decision making, and serves as a reference point for others who are selecting LLM judges.

Introduction

Humanity’s Last Exam (HLE) is a diverse question-answer benchmark. Due to its simplicity and persistent difficulty, it’s among the most popular benchmarks in Inspect Evals.

There are two question types: multiple-choice, or open-ended (referred to as “exact match”).¹

categoryScores = FileAttachment("data/category_scores.json").json()
hle.hleCategoryBars(null, categoryScores, {
  caption: html`GPT-5.6 on the 668 questions of
    <a href="https://arxiv.org/pdf/2602.13964v2">HLE Verified Gold</a>. Its score
    ranges from 24% on Biology/Medicine to 59% on Math.`,
})

The scoring uses an LLM judge for grading; the default LLM judge is o3-mini-2025-01-31, which OpenAI is deprecating on October 23rd, 2026.

glr.mount(glr.components.TranscriptExample, {
  src: "protected/grader_prompt.json",
  title: "The prompt given to the judge",
  description: "",
  buttonLabel: "Reveal",
  showCount: false,
})

Methodology

How we tested judge candidates

We generated answers to HLE questions with Opus 5 and GPT-5.6. Then we judged these answers using models on the chart below via OpenRouter. We used a panel of three frontier models — Opus 5, GPT-5.6 and Gemini 3.1 Preview — as pseudo-ground-truth.

Pseudo-Ground-Truth

We used a panel of three frontier models from three different providers — Opus 5, GPT-5.6 and Gemini 3.1 Preview — as pseudo-ground-truth. On the vast majority of questions these judges were in agreement, but 112 out of 4,943 decisions ended up contested. To disambiguate these, we ran a deliberation pipeline on these 112 questions. On 77 of the 112, the judges reached consensus, on the remaining 35 we chose the 2-out-of-3 simple majority decision.²

Pricing on Openrouter

We ran the judges between August 3rd and September 3rd. All models were accessed via OpenRouter, where the cost per mtok varies across providers and time. We let OpenRouter determine the provider and the unit pricing using their algorithm.

If a model was on promotion or its cost later changed, we didn’t do any price adjustment.³

Results

Most judging is easy but models aren’t hitting a ceiling

There is a high level of agreement between judge verdicts and this pseudo-ground-truth — 96%+ even for small open-source models such as gpt-oss-20b. This is because judging most questions doesn’t require much intelligence: models just have to do fuzzy matching on extracted answers.

But models also aren’t hitting an “intelligence ceiling” for judging this eval yet — more money can give you better accuracy.

The long tail of questions that are hard-to-judge

There is a long tail of questions that require intelligence to judge. Here is one that Opus 5 gets right but other models get wrong.

Using a more advanced model gets you more accurate judgements on questions like this one.

Accuracy impact on the final score

A worse judgement does not necessarily translate into a lower score.

For example, the final score when using o3-mini is almost the same as when using Claude Opus 5. This is because judges have biases. In case of o3-mini, it’s more lenient when it comes to judging small numerical differences. A numerical gap that would lead to Opus 5 declaring the answer incorrect is often tolerated by o3-mini.

The new grader for Inspect Evals

In the default configuration we ended up replacing o3-mini with a set of two judges:

1. GLM 5.3, the best Pareto frontier judge among models cheaper and better than o3-mini.
2. Gemini 3.7 Flash, as a backup judge that’s close to the frontier.

The decision ultimately required some intuitive judgement, which is a substituting for a lack of validated algorithmic methods for selecting optimal judges. Our selection criteria are described in the footnotes.⁴

We chose Gemini-3.7 Flash as the backup model because:

• Its cost is comparable to o3-mini, but it aligns better with our accuracy measure.
• It’s from a well established AI lab (“Nobody Ever Got Fired for Buying IBM”).
• It’s likely to be available for a long time, probably for ≈1.5 years⁵
• It’s very fast, reducing the likelihood that the second judge will extend the wall clock time for the eval run.

Footnotes: