How we benchmark models on llm.blog

Every number on the leaderboard comes out of the process on this page. When this page and a leaderboard cell disagree, the cell’s measurement date tells you which is newer; the methodology described here is versioned, and harness changes are logged in the changelog with dates.

The five rules

Our measurement policy compresses to five rules:

1. Vendor-default settings. We run each model the way a developer gets it from the API with no tuning: default temperature, default reasoning effort, vendor-recommended system prompt if the docs specify one. Benchmarks are meant to predict what you get, not what is achievable.
2. Fixed public scaffold. Every model runs the same agentless-lite scaffold for SWE-bench Verified and the official containers for Terminal-Bench. Scaffold code is pinned by commit hash and linked from each measurement cycle’s notes.
3. Three seeds, median reported. Single-run benchmark scores are noise wearing a suit. We run three seeds and report the median; the spread is retained internally and cited when a movement is within variance.
4. Every score carries a date. Serving-stack revisions move scores without version-number changes. A score without a measurement date is not a fact; it is a rumor with decimals.
5. Unreproduced numbers are flagged. Where we display a vendor-published figure we have not reproduced — new model, new benchmark, harness gap — it renders with an explicit unverified flag until we can measure it ourselves.

Benchmark selection

We track six benchmarks. The selection favors measures that predict production behavior for API practitioners over leaderboard-culture staples:

Benchmark	What it predicts	Known weakness
SWE-bench Verified	Repository-scale coding agents	Python-heavy; scaffold-sensitive
Terminal-Bench 2.0	Operational/terminal competence	Young benchmark, small task set
GPQA Diamond	Expert reasoning ceiling	Saturating at the frontier
AIME 2025	Structured mathematical reasoning	Contamination pressure grows yearly
MMMU	Multimodal understanding	Weak proxy for document extraction
LiveCodeBench	Contamination-controlled coding	Contest style ≠ production code

Two absences are deliberate. We do not track MMLU or its descendants: at frontier accuracy the residual questions are disproportionately mislabeled or ambiguous, so movement measures label noise. We do not track head-to-head preference Elo (arena-style rankings): preference voting measures persuasiveness under casual prompts, which diverges from task competence in ways that reward stylistic tuning.

Contamination policy

Contamination is not a scandal; it is a baseline condition to engineer around. Our posture:

• Prefer benchmarks with rolling post-cutoff problem sets (LiveCodeBench) or verified human curation (SWE-bench Verified).
• Track year-stamped sets (AIME 2025) only while the newest models’ training cutoffs plausibly precede the problems, and retire them after.
• Watch for the contamination signature — a model dramatically outperforming its own capability profile on one aged benchmark — and annotate rather than delete when we suspect it.

Variance and what counts as movement

Run-to-run spread on agentic benchmarks is larger than most coverage admits: ±0.4 points is typical on SWE-bench Verified at three seeds, and Terminal-Bench 2.0 runs wider. Our reporting rule: a month-over-month delta inside the observed seed spread is described as “stable,” not as movement. Deltas beyond spread get called movement and, where we can identify it, a cause — serving update, harness revision, or unexplained.

When the harness itself changes — container image updates, scaffold pin bumps — we re-run affected benchmarks for all models in the same cycle, never just the new arrivals, and note the change in the cycle notes. A harness change that moves any median more than 0.5 points triggers a full historical annotation.

What the leaderboard is not

The leaderboard ranks measured capability on six axes. It does not rank:

• Value. Price-per-capability is workload-specific arithmetic; the comparison pages do that math per pairing, and the calculator on each one does it for your token mix.
• Safety or alignment properties. We do not have the harness to measure them credibly, and we decline to launder vibes into a column.
• Latency and throughput. Serving performance varies by region, tier, and hour; point-in-time measurements would mislead more than inform. We may add percentile-tracked latency in future with continuous measurement.

Reproducing our numbers

Scaffold commit hashes, container digests, seed values, and per-task result JSON for each cycle are retained. The dataset behind the leaderboard — scores, dates, sources, flags — is the same JSON that renders the site, and the llms.txt file points machine readers at the canonical URLs. Cite the leaderboard with its measurement date and the numbers will stay attributable even after later refreshes move them.

Corrections: [email protected]. A correction that survives verification lands in the next cycle notes with credit.