Auditing BixBench - Generality Labs

glVersion = "dev"
gl = import(`/assets/gl-components/gl.js?v=${glVersion}`)
mc = import(`/assets/gl-components/model-colors.js?v=${glVersion}`)
bx = import(`/blog/posts/bixbench-broken-ceiling/components.js?v=${glVersion}`)

We’re presenting the results from our recent audit of BixBench (Bioinformatics Benchmark), an agentic benchmark which is used to evaluate dual-use scientific capabilities.
At first glance, BixBench is far from saturated. However, we find that two issues (flawed tasks and judge grading errors) explain roughly 70% of failed submissions. Correcting for them can increase measured agent performance by up to 42 percentage points.
The full audit report, which provides additional details, and our summary scorecard, which gives a quick overview, are available here.

What is BixBench?

BixBench is an agentic QA benchmark designed to evaluate whether agents can perform realistic, open-ended bioinformatics analyses.

BixBench was released by FutureHouse in February 2025, and results have been reported by OpenAI, Anthropic, and in Stanford’s AI Index Report. The dataset has over 38k downloads as of August 2026.

Eval architecture overview

For a given BixBench task, the agent receives a system prompt and a task-specific user prompt containing the research question and instructions for how to proceed. The agent undertakes its analysis in a Jupyter notebook, running code on the task’s provided data files, and ends the task by submitting a short-form answer. A single judge compares the submission to the reference answer and grades it as correct, incorrect, or a refusal by the model.

// Architecture diagram. All wording is here: edit the strings, then
// `make render`. `note` on a file or the notebook becomes a hover tooltip.
bx.bixArchitecture(null, {
  prompt: { title: "Prompt",
    items: [
      { label: "System prompt" },
      { label: "User prompt template", children: [
          { label: "Question", vary: true },
          { label: "Instructions" },
        ] },
    ] },
  agent: { title: "Agent",
    tools: ["list_workdir()", "edit_cell()", "submit_answer()"] },
  env: { title: "Environment",
    notebook: { label: "notebook.ipynb", sub: "Jupyter notebook",
      sub2: "(starts blank)",
      note: "The agent writes and runs cells here. The researchers' original analysis notebook is withheld." },
    filesLabel: "Data files",
    files: [
      { ext: ".csv",  vary: true, note: "comma-separated data tables" },
      { ext: ".xlsx", vary: true, note: "Excel spreadsheets" },
      { ext: ".txt",  vary: true, note: "plain-text tables, for example gene count matrices" },
      { ext: ".faa",  vary: true, note: "protein sequences in FASTA format" },
      { ext: ".fastq.gz", vary: true, note: "compressed sequencing reads" },
      { ext: ".zip",  vary: true, note: "archives, for example BUSCO result bundles" },
    ],
    callLabel: "Tool calls", returnLabel: "Outputs" },
  grader: { title: "Grader",
    submitted: "Submitted answer",
    inputs: [
      { label: "Grader prompt" },
      { label: "Question", vary: true },
      { label: "Reference answer", vary: true },
    ],
    outcomes: [{ text: "Correct", tone: "good" },
               { text: "Incorrect", tone: "bad" },
               { text: "Refusal", tone: "neutral" }] },
}, {
  legend: [
    { label: "Varies per sample", vary: true },
    { label: "Same for every sample", vary: false },
  ],
})

Dataset

BixBench’s dataset is built from a ground-truth dataset of 61 bioinformatics research settings (i.e. self-contained research projects), which were sourced from “human experts”. Settings contain hypotheses, datasets, Jupyter notebooks with data analysis, and research conclusions.

To construct the benchmark questions, the authors used Claude Sonnet 3.5 to generate 488 candidate questions based on the research settings. Reviewers then filtered this down to a final dataset of 205.

Each rollout begins with a user prompt that includes one of these 205 questions, along with various instructions. The environment contains a blank notebook (the original notebook is withheld), and files from the corresponding research setting.

glr.mount(glr.components.TranscriptExample, {
  src: "protected/example_transcript.json",
})

Solver

The agent is given 40 turns to complete the analysis and submit its answer. The authors use a custom agent harness with access to three tools:

list_workdir() for file discovery
edit_cell() for editing and running notebook cells
submit_answer() for finishing the task

Although the original implementation is notebook-centric, it’s possible for coding agents to complete the tasks without using notebooks.

Scorer

When the agent submits its response, a single LLM judge (GPT-4o) compares the answer to a reference solution.¹

// Original BixBench OPEN_ENDED_GRADING_PROMPT, gated like the episode prompt.
glr.mount(glr.components.TranscriptExample, {
  src: "protected/grader_prompt.json",
  title: "BixBench's Grader Prompt",
  description: "The grader prompt is gated, click to reveal.",
  buttonLabel: "Reveal",
  showCount: false,
})

Frontier models fail half of BixBench

For this audit, we evaluated six different models on all 205 questions, using various Inspect implementations.²

On a first pass, BixBench looks genuinely challenging, with frontier models solving only ~50% of the questions.

Our cumulative success plot below suggests that this ceiling is largely insensitive to harness, with neither Claude Code nor Codex lifting it. We also find that the turn limit is not a major constraint, with only 1.5% of episodes terminating because they reach it.

infData = FileAttachment("data/inference_scaling.json").json()
evalLogs = FileAttachment("data/eval_logs.json").json()
gl.glInfSteps({Plot}, infData, {
  track: "official",
})

Given that similar benchmarks such as CORE-Bench are already saturated, this consistently lower performance on BixBench is surprising.

BixBench’s misfiring judge

The first notable error with BixBench is a faulty judge. By default, BixBench employs GPT-4o to grade submissions and provides it with the question, the reference answer, and the short-form answer submitted by the agent. It dutifully obeys its instruction to output a single binary score with no reasoning; however, this leads to a false negative rate of almost 10% and a false positive rate of 1.6%.³

Analysing the transcripts, we grouped the judge’s failure modes into four categories. The donut chart shows the percentage of submissions misgraded by each failure mode, and examples are provided in the interactive below.⁴

judgeBreakdown = FileAttachment("data/judge_error_breakdown.json").json()
// one colour per error pattern, shared with the interactive's pills below.
// The blue was moved to a cyan: against the purple it failed both the
// colour-blind and the normal-vision separation checks.
misgradeColors = ({
  "answer within range": gl.palette.bad,
  "rounding / precision": "#7c4dbe",
  "units / format": "#0e8ea8",
  "wrong answer accepted": gl.palette.warn,
})
// The category strings are shared with the interactive below and with the
// data files, so they are capitalised for display only.
sentenceCase = (s) => s.charAt(0).toUpperCase() + s.slice(1)
// Every percentage on the donut — ring labels and the key on the right — is
// that category's share of ALL judged submissions (1,638), i.e. the
// percentage of submissions misgraded with that failure mode, so the four
// sum to the centre's overall misgrade rate (6.1%). Raw counts live in the
// hover; the direction-specific rates (9.2% and 1.6%) in the prose above.
// Largest-remainder rounding, so the one-decimal ring labels sum exactly to
// the one-decimal centre figure.
misgradeShares = {
  const raw = judgeBreakdown.rows.map(r => 1000 * r.gradings / judgeBreakdown.judgedGradings);
  const out = raw.map(Math.floor);
  let short = Math.round(raw.reduce((a, b) => a + b, 0)) - out.reduce((a, b) => a + b, 0);
  const order = raw.map((v, i) => [v - Math.floor(v), i]).sort((a, b) => b[0] - a[0]);
  for (const [, i] of order) { if (short-- <= 0) break; out[i] += 1; }
  return new Map(judgeBreakdown.rows.map((r, i) => [r.label, out[i] / 10]));
}
gl.glPie(judgeBreakdown.rows.map((r) => ({
  label: sentenceCase(r.label),
  // value drives both the geometry and the key figure on the right; using
  // the rounded share keeps ring and key showing the identical number
  value: misgradeShares.get(r.label),
  ring: `${misgradeShares.get(r.label).toFixed(1)}%`,
  gradings: r.gradings, questions: r.questions,
})), {
  colorDomain: Object.keys(misgradeColors).map(sentenceCase),
  colorRange: Object.values(misgradeColors),
  ringFmt: (d) => d.ring,
  countFmt: (d) => `${d.gradings} answers across ${d.questions} question${d.questions === 1 ? "" : "s"}`,
  // key shows swatch + label only; the percentages live on the ring
  keyVal: false,
  // centre: the overall judge error rate — 100 misgrades out of the 1,638
  // judged gradings (both directions pooled)
  total: `${(100 * judgeBreakdown.gradings / judgeBreakdown.judgedGradings).toFixed(1)}%`,
  totalSub: ["of submissions", "were misgraded"],
  className: "gl-judge-bars", margin: ".45rem 0", center: true,
})

judgeIndex = FileAttachment("data/judge_misgrades_index.json").json()
glr = import(`/assets/gl-react.js?v=${glVersion}`)
glr.mount(glr.components.JudgeErrors, {
  src: "protected/judge_misgrades.json", index: judgeIndex, colors: misgradeColors,
})

Over a third of BixBench’s tasks are broken

Probing the model transcripts surfaces additional issues.

Looking at the submitted answers, we found that frontier models independently converged on the same incorrect answers for 35% of tasks — a suspicious signal. In auditing each of BixBench’s 205 tasks, we found that over a third are problematic: 20% have reference answers that cannot be defensibly reproduced from the supplied data, while a further 17% depend on unspecified analytical choices.

sankeyData = FileAttachment("data/failure_sankey.json").json()
bx.bixFailureSankey(null, sankeyData, {})

Measured agent performance can rise by 42 percentage points

Taking into account BixBench’s faulty judge and broken samples reveals its false ceiling.

gl.glInfSteps({Plot}, infData, {
  track: "official",
  bands: [
    { from: infData.meta.solvableCeil, to: 1, hatch: true, color: gl.palette.bad,
      label: `${infData.meta.nBroken} of ${infData.nTasks} samples (38%) are broken` },
    { from: infData.meta.judgeFloor, to: infData.meta.solvableCeil, hatch: true, angle: -45, color: gl.palette.warn,
      label: "the judge incorrectly grades up to 11% of a run's samples" },
  ],
})

After removing these broken samples and correcting judge errors, we find that measured agent performance can rise by as much as 42 percentage points.

scoreSteps = FileAttachment("data/score_adjustments.json").json()
bx.bixScoreSteps({Plot}, scoreSteps, {})

And by replotting our pass@1 cumulative success results, we see that BixBench is effectively saturated.

// One colour per model (matching the plots above); solid = the same runs'
// official scores on default BixBench (all 205 tasks, stock judge),
// dashed = judge-corrected grades on the 128 non-broken tasks.
gl.glInfSteps({Plot}, infData, {
  track: "solvable",
  suffix: " (corrected)",
  dash: "6,4",
  models: ["opus5", "kimik3", "gpt56sol"],
  compare: { track: "official", suffix: " (default)", dash: "" },
})

Wait, how did OpenAI report GPT-5.5 as scoring 81% on BixBench?

GPT-5.5’s reported score of 80.5% on BixBench is difficult to reconcile with our results on the original 205-question benchmark. So what explains the gap?

One plausible explanation is that OpenAI used the BixBench-Verified-50 variant. BixBench-Verified-50 was released by Phylo, an AI-for-biology research lab, in February 2026 after its own investigation surfaced similar problems, including “ambiguous or underspecified questions and incorrect ground truths”⁵.

Reassuringly, the overlap with our independently conducted audit is good, with 80% of the verified subset agreeing with our audit results⁶.

Conclusion

Our audit surfaced two substantial issues: over a third of BixBench’s tasks are not solvable as given, and the default judge incorrectly grades up to 11% of samples in our runs. Consequently, we have graded BixBench as a C-tier eval under Generality Labs’ auditing framework: it has notable problems, but contains some useful signals.

For prospective users, our minimum recommendations are:

Use BixBench-Verified-50 rather than the original dataset.
Patch the grader numerical range handling and upgrade the grader model⁷.

Lastly, the .eval logs used in this audit are available on request.

Footnotes

The original paper reports results with Claude Sonnet 3.5 as the judge, although with Sonnet 3.5 now deprecated, the authors’ GitHub repository uses GPT-4o.↩︎
For a majority of the runs, we used a community-contributed replication by Concordia AI. For the Claude Code and Codex runs, we used separate task variants built on Inspect’s sandbox_agent_bridge, which run each vendor’s CLI inside the task sandbox (the same approach as Inspect SWE). We also used Concordia’s 80-turn limit rather than the original’s 40-turn limit — these are roughly equivalent as a tool result costs a message in Inspect.↩︎
The judge audit covers 1,640 gradings: 205 questions × 8 runs (six models, plus the Claude Code and Codex harness variants; two unscored Codex samples never reached the judge). 969 of those gradings were marked incorrect, and 89 of them were correct answers (i.e. false negatives), giving 9.2%. Of the 669 answers marked correct, 11 were actually wrong (i.e. false positives), giving 1.6%. GPT-4o is the weakest model we evaluated and accounts for a sixth of the incorrect gradings on its own, so including it lowers the false negative rate; across the other runs it is 10.4%.↩︎
Note for “Answer within range” failures: 61 of the 205 questions have a numerical range as the reference answer (e.g. 0.74,0.77). The BixBench dataset clearly labels these questions with eval_mode = "range_verifier", and the original code contains specific logic for grading numerical range answers, including a numeric bounds check, and a judge prompt explaining the range format. However, two implementation oversights (1, 2) mean that neither is applied when grading agents’ answers. In practice, both the original grader and the Inspect implementation we evaluated show the judge the raw range text without telling it that values inside the range count as correct.↩︎
BixBench-Verified-50 was created by domain experts independently re-reviewing a subset of 50 BixBench tasks. 17 of these 50 received revised question text, corrected ground-truth answers, or both.↩︎
10 of the tasks our audit flagged as flawed are rephrased or corrected in the verified subset, while 3 were kept verbatim. 30 we didn’t flag were kept verbatim, while a further 7 were rephrased or corrected.↩︎
Fixing the scorer logic to use the intended prompt (OPEN_ENDED_RANGE_GRADING_PROMPT) would be a good start.↩︎