News · benchmarks · methodology · terminal-bench
Terminal-Bench 2.0 joins the llm.blog tracked benchmark set
Draft — pending editorial review
llm.blog added Terminal-Bench 2.0 to its tracked benchmark set on July 15, 2026, scoring six models in the first cycle. Claude Opus 4.5 leads at 59.3%. Terminal-Bench measures task completion in a live terminal environment — operational competence that SWE-bench's repository-patching format does not capture.
We added Terminal-Bench 2.0 to the tracked benchmark set on July 15, 2026, and scored six models in the first measurement cycle: Claude Opus 4.5 (59.3%), Gemini 3 Pro (54.2%), Claude Sonnet 4.5 (50.0%), Kimi K2 Thinking (47.1%), Claude Haiku 4.5 (41.0%), and Grok 4.1 (pending a harness fix). Remaining models join in August. All scores and dates are on the leaderboard.
Why Terminal-Bench
Terminal-Bench drops a model into a live terminal and scores task completion: environment setup, debugging, data wrangling, service configuration. It measures operational competence — the “can it actually drive a computer” question — where SWE-bench Verified measures repository-scale code modification. The two disagree in informative ways: models tuned aggressively for patch generation can underperform on open-ended terminal tasks, and the gap predicts real-world agent reliability better than either number alone.
Configuration
We run the standard 2.0 task set in the official container, vendor-default reasoning settings, 20-minute per-task timeout, two attempts with the better run discarded (we report first-attempt scores; the second run measures variance only). Full harness details are in the methodology, which has been updated with the Terminal-Bench section as of July 15, 2026.
Vendor-published Terminal-Bench numbers exist for several tracked models and differ from ours by up to four points, mostly attributable to scaffold differences. Model pages flag vendor figures unverified where we have not reproduced them.