llm.blog — LLM news, benchmarks, and engineering analysis

llm.blog dataset refresh: 14 current models, AA Intelligence Index adopted

Tue, 11 Aug 2026 12:00:00 GMT

llm.blog rebuilt its model dataset against Artificial Analysis on August 11, 2026: 14 current models with verified pricing and the AA Intelligence Index as primary metric. Claude Opus 5 leads at 63, Claude Fable 5 follows at 62, Kimi K3 tops open weights at 60, and DeepSeek V4 Flash holds the price floor at $0.14 per million input tokens.

We rebuilt the llm.blog model dataset on August 11, 2026, against Artificial Analysis — replacing the prior tracked set, whose specifications and prices no longer reflected the market. Fourteen current models now carry verified per-MTok pricing, context windows, and the AA Intelligence Index as the leaderboard's primary metric, each value dated to the day we verified it.

The state of the frontier, measured

The top of the index as of this refresh:

Model	Maker	Index	$/MTok in · out
Claude Opus 5	Anthropic	63	$5 · $25
Claude Fable 5	Anthropic	62	$10 · $50
Kimi K3	Moonshot	60	$3 · $15
GPT-5.6 Sol	OpenAI	61	$5 · $30
Qwen3.8 Max	Alibaba	58	$2 · $6

Three facts stand out. Claude Opus 5 leads the index at 63 while undercutting its own premium sibling — Fable 5 costs twice as much and measures a point lower. Kimi K3, a 2.8-trillion-parameter open-weights MoE, sits three points off the closed frontier — the narrowest open-closed gap yet measured. And DeepSeek V4 Flash at $0.14/$0.28 per million tokens delivers an index score of 52 — matching GPT-5.6 Luna and Gemini 3.6 Flash — at prices Artificial Analysis rates the lowest of any well-known model.

What changed methodologically

The AA Intelligence Index — an independent composite of ten evaluations — is now the leaderboard's primary column, recorded with the date we verified it rather than presented as our own measurement. Columns from our own harness (SWE-bench Verified, Terminal-Bench) will join as those runs land; the harness protocol is unchanged in the methodology. Pricing now cites each model's Artificial Analysis page as of the verification date.

The full table, sortable with per-cell dates, is on the leaderboard. All fifteen head-to-head comparisons were rewritten against the new data the same day.

Meta's Muse Spark 1.2 and Alibaba's Qwen3.8 Max land in the frontier top ten

Tue, 11 Aug 2026 12:00:00 GMT

Meta released Muse Spark 1.2 on August 5, 2026 — a proprietary multimodal model scoring 57 on the Artificial Analysis Intelligence Index at $1.25 per million input tokens. Alibaba's Qwen3.8 Max, released August 3, scores 58 at $2 input and $6 output — the lowest-priced flagship tier of any tracked frontier model.

The first week of August delivered two frontier entrants within three days of each other, both measured by Artificial Analysis and both now on the leaderboard.

Muse Spark 1.2: Meta's proprietary turn

Muse Spark 1.2, released August 5, 2026, scores 57 on the AA Intelligence Index — ahead of every previous Meta model — with text, image, speech, and video input at $1.25 per million input tokens and $4.25 per million output. The strategic fact is bigger than the score: this is Meta shipping a proprietary frontier model rather than open weights, a reversal of the Llama-era posture that defined the company's AI positioning. Its serving performance is not yet benchmarked, and the API ecosystem around it is days old.

Qwen3.8 Max: flagship scores at mid-tier prices

Qwen3.8 Max, released August 3, 2026, scores 58 — fourth among all tracked models — at $2 per million input tokens and $6 per million output, with text, image, and video input. No other model above 55 on the index prices output under $10; Qwen prices it at $6. In our head-to-head against GPT-5.6 Sol, that works out to 95% of the flagship measurement at roughly a fifth of the blended cost.

The pattern

Both entrants attack the same seam: the gap between mid-tier prices and flagship capability. With Gemini 3.6 Flash's output cut to $7.50 and DeepSeek V4 Flash anchoring the floor at $0.28, the price of measured intelligence continues to fall on every tier — the through-line of 2026's API market. Every figure above carries its measurement date on the relevant model page; the dataset refresh that accompanies these additions is documented in today's refresh note.

How we engineer résumés that survive the ATS at F1Jobs.io

Tue, 11 Aug 2026 12:00:00 GMT

F1Jobs.io is NeuraScribe's career-acceleration service for international students in the United States. Human recruiters and LLM pipelines rebuild each résumé to survive applicant tracking systems — clean parsing, targeted keywords, no gimmicks — then run applications, interview preparation, and mock interviews. This piece documents how the ATS actually reads a résumé and how we engineer for it.

Every job application at a mid-size or large US employer passes through an applicant tracking system before it reaches a person. For international students — filtered by visa questions, squeezed by OPT timelines, and often carrying résumés formatted for another country's conventions — the ATS is where most applications quietly end. At F1Jobs.io, the NeuraScribe company built for exactly this population, résumé engineering for the ATS is the first thing we do for every candidate, and it is the reason the rest of the pipeline works.

This is a first-party piece: F1Jobs.io is our product. What follows is how the system actually works, because the mechanics are the argument.

What an ATS actually does with a résumé

The folk model of the ATS — a robot that reads your résumé and rejects it — is wrong in a useful way. What actually happens is closer to an ETL pipeline:

Parsing. The document is converted into structured fields: contact block, work history entries with dates and titles, education, skills. Parsing quality is the single biggest hidden variable. Two-column layouts, text inside tables or images, headers and footers, and decorative section names ("Where I've Made Impact") all degrade extraction — and a field the parser missed is a field that does not exist.
Enrichment and matching. Parsed fields are normalized (titles mapped to canonical roles, skills to taxonomies) and scored against the job requisition. Modern systems increasingly use embedding similarity alongside keyword matching, but the job description's own vocabulary remains the strongest signal.
Knockouts and ranking. Application questions — work authorization, sponsorship requirements, years of experience — act as hard filters. Everything that survives is ranked into the queue a recruiter actually reads, usually top-down under time pressure.

Nothing in that pipeline reads for elegance. It reads for extractability and relevance. That is an engineering target.

The F1Jobs.io résumé pipeline

Our pipeline is hybrid by design: LLMs do structure and coverage, recruiters own judgment and truth.

Parse-first formatting. Every résumé is rebuilt into a single-column, semantically-headed layout that parses cleanly in the systems that matter. We verify by round-tripping: parse the rendered document with the same class of extraction tooling the ATS uses and diff the structured output against ground truth. If a job entry loses its date range in the round trip, the layout is wrong, whatever it looks like.

Requisition-driven targeting. For each application, an LLM pass aligns the résumé's vocabulary with the specific posting — the titles, skills, and phrasing the requisition itself uses — constrained to facts the candidate has verified. This is the line between optimization and fabrication, and it is enforced by a human recruiter, not a system prompt: nothing ships that the candidate cannot defend in an interview.

Knockout strategy. Visa and authorization questions are answered accurately and strategically — accurately, because lying to a knockout filter wastes everyone's time including the candidate's; strategically, because which roles to apply to at all is where recruiters earn their keep. A candidate with 18 months of OPT runway should not spend it on requisitions whose filters were always going to fire.

Execution and preparation. Résumé engineering feeds the rest of the service: recruiters run the application volume, and the interview side — preparation dossiers, mock interviews, positioning of LinkedIn and GitHub profiles — takes over for everything that survives the funnel. Candidates have landed interviews across technology, finance, automotive, and enterprise employers on this pipeline.

What we refuse to do

ATS folklore is full of tricks that used to circulate on forums: white-text keyword blocks, invisible fonts, stuffing the skills section with every framework ever released. We don't do any of it. Parsers flag hidden text, duplicate-keyword density reads as spam, and every trick that survives the parser still has to survive the recruiter reading the ranked queue. The durable edge is boring: clean structure, honest content, requisition-specific vocabulary, and enough application volume executed consistently. Machines read first; people decide. Engineer for both, in that order.

Why we built our own ATS: inside the NeuraScribe staffing OS

Tue, 11 Aug 2026 12:00:00 GMT

NeuraScribe is the white-label operating system for staffing companies: applicant tracking, recruiter CRM, onboarding, e-signature contracts, and billing in one platform that replaces more than eight disconnected tools. Bootstrapped to $1M ARR within its first year, it exists because legacy ATS products were built for corporate HR, not staffing firms. Here is the architecture.

NeuraScribe exists because of a category error in recruiting software. The applicant tracking system — the ATS — was designed for corporate HR: one employer, its own requisitions, a pipeline that ends at the hire. Staffing companies and consultancies took those tools because nothing else existed, then spent a decade duct-taping CRMs, spreadsheets, e-signature tools, invoicing software, and payroll exports around them. The average firm we talk to runs more than eight disconnected systems, and the thing none of them can answer is the business's only real question: which recruiter earned this revenue?

This is a first-party piece — NeuraScribe is our product, and I run its engineering. What follows is the architecture and the reasoning, because staffing operators evaluating an ATS deserve the mechanics, not adjectives. The short version of the traction: bootstrapped, no outside capital, and past $1M in annual recurring revenue inside the first year.

The staffing business is an attribution problem

A corporate ATS models a funnel. A staffing firm is a marketplace with three sides — clients, candidates, recruiters — and its unit economics live in the joins. The same candidate can be in play for three clients at once. Two recruiters can touch the same placement: one sourced, one closed. The invoice for a contract placement has to reconcile against timesheets, the recruiter's commission against the invoice, the firm's margin against both.

So the core of the NeuraScribe staffing OS is not the pipeline view — every ATS has a pipeline view. It is the attributed event ledger underneath it: every application, interview, submission, contract, and invoice is a record that carries the recruiter who earned it. Commissions, leaderboards, margin reports, and client statements are all projections of the same ledger. When firms ask what "one live source of truth" means in practice, that ledger is the answer — data pipelines that think, pointed at a business that runs on attribution.

One system where eight used to be

The platform ships as a single white-label system under the firm's own brand:

Applicant tracking and recruiter CRM — clients, requisitions, candidates, and pipelines in one relational model, not synced across two products with two truths.
Candidate onboarding — document collection, verification, and portal access, branded as the firm.
Contracts and e-signature — offers and agreements generated from the same records they bind, signed in-flow, attributed like everything else.
Billing and team management — invoices reconciled to placements, commissions computed from the ledger, roles and permissions per desk.

White-label is not cosmetic. A staffing firm's asset is its client relationships; software that inserts its own brand between firm and client is charging rent on someone else's asset. Candidate portals, dashboards, offer letters, and invoices all carry the firm's identity — NeuraScribe is the machinery underneath.

Where the AI actually lives

The industry's default is a chatbot bolted onto a legacy ATS. We built workflows into the pipeline instead, each writing back into the attributed ledger:

The interview engine generates a branded, seven-section preparation dossier for each candidate-and-role pairing — company context, role analysis, likely interview structure, and preparation strategy — the artifact recruiters used to spend hours assembling by hand, produced in minutes and reviewed before it ships.
Résumé optimization rebuilds candidate documents for ATS parseability and requisition-specific vocabulary — the same discipline documented in our F1Jobs.io piece.
Matching and profiling score candidates against open requisitions across every client the firm serves, so a candidate rejected by one client surfaces for the next instead of dying in a silo.

The design rule for all of it: AI drafts, humans own. Every generated artifact has a recruiter accountable for it, and every action lands in the same ledger the business is billed from.

What we learned bootstrapping it

Building this without outside capital forced one discipline that shaped the product more than any roadmap: every feature had to be something a staffing firm would pay for this quarter, not something a category analyst would praise next year. That filter killed dashboards nobody reads and funded the unglamorous machinery — contract generation, invoice reconciliation, commission math — that actually replaces the eight tools. It is also why the platform sells as an operating system rather than a point solution: the checkbook only opens when the whole stack collapses into one bill.

Staffing founders evaluating the switch can reach us at neurascribe.org. Engineers curious about the extraction and verification patterns behind the AI workflows: the structured-output verification deep dive is the same playbook we run in production here.

Google cuts Gemini 3 Flash input pricing 17% to $0.25 per million tokens

Mon, 10 Aug 2026 12:00:00 GMT

Google cut Gemini 3 Flash input pricing from $0.30 to $0.25 per million tokens on August 10, 2026, a 17% reduction. Output pricing holds at $2.50 per million. The cut matches GPT-5 mini on input cost while offering a context window 2.5 times longer at 1M tokens.

Google reduced Gemini 3 Flash input pricing to $0.25 per million tokens on August 10, 2026, down from $0.30. Output pricing is unchanged at $2.50 per million tokens. The change applies immediately across the Gemini API, Vertex AI, and AI Studio, and appears in Google's pricing documentation.

This is the second Flash-tier reduction of 2026. The move puts Gemini 3 Flash at exact input-price parity with OpenAI's GPT-5 mini ($0.25 per million tokens, priced August 2025) while carrying a 1,048,576-token context window against mini's 400,000.

What changed

Metric	Before	After	Δ
Input, per MTok	$0.30	$0.25	−17%
Output, per MTok	$2.50	$2.50	—

Cached-input and batch pricing scale from the new base rate. Google did not announce corresponding cuts for Gemini 3 Pro.

Why it matters

Input tokens dominate cost in retrieval and summarization workloads, where prompts routinely run 50–100× longer than completions. At the new rate, a pipeline processing 10B input tokens monthly saves $500 per month — small per-workload, decisive at platform scale, and a direct answer to the volume-tier pricing war that began with DeepSeek-V3.2's sparse-attention cuts in September 2025.

Our Gemini 3 Flash vs GPT-5 mini comparison and the pricing columns on the leaderboard reflect the new rate as of August 10, 2026.

Correction policy: pricing figures are re-verified against vendor documentation on every data refresh; this post's figures were last checked August 10, 2026.

Anthropic promotes Claude Opus 4.5 snapshot 20260805 to API default

Sat, 08 Aug 2026 12:00:00 GMT

Anthropic promoted snapshot claude-opus-4-5-20260805 to the API default for Claude Opus 4.5 on August 8, 2026. The update tightens tool-call formatting compliance and fixes a long-context truncation edge case. Anthropic reports unchanged benchmark scores. Requests pinned to the prior snapshot continue to resolve until deprecation notice.

Anthropic switched the default resolution of claude-opus-4-5 to snapshot claude-opus-4-5-20260805 on August 8, 2026, per the model documentation. Requests that pin the previous snapshot explicitly continue to resolve; Anthropic's deprecation policy provides notice before any snapshot retirement.

What changed

The 20260805 snapshot addresses two production behaviors:

Tool-call formatting compliance. The update reduces malformed JSON in parallel tool-call blocks under long system prompts — a failure mode agent framework maintainers have tracked since June.
Long-context truncation. A fix for an edge case where responses near the 64K output ceiling could terminate mid-token sequence when streaming with stop sequences configured.

Anthropic states benchmark scores are unchanged from the launch measurement (80.9% SWE-bench Verified, November 24, 2025). We will re-measure in the September leaderboard refresh and note any drift on the Claude Opus 4.5 model page.

Practitioner notes

Teams pinning snapshots for reproducibility should schedule validation against 20260805 before the prior default enters deprecation. Teams resolving the bare claude-opus-4-5 alias received the new snapshot automatically on August 8. No pricing or rate-limit changes accompany this promotion; current pricing remains $5 input / $25 output per million tokens (verified August 8, 2026).

The full version history for this model is on its changelog section.

August refresh: all 12 tracked models re-scored on SWE-bench Verified

Wed, 05 Aug 2026 12:00:00 GMT

llm.blog re-measured all 12 tracked models on SWE-bench Verified during the first week of August 2026. Claude Opus 4.5 leads at 80.9%. Gemini 3 Flash posted the cycle's largest gain. Every score on the leaderboard now carries an August 5, 2026 measurement date and harness configuration notes.

We completed the August SWE-bench Verified refresh on August 5, 2026, re-running all 12 tracked models through the harness described in our methodology. Every leaderboard cell now carries this cycle's measurement date.

Results snapshot

The ordering at the top is unchanged from July: Claude Opus 4.5 leads at 80.9%, with Claude Sonnet 4.5 second at 77.2%. Movement concentrated in the middle of the table:

Gemini 3 Flash posted the cycle's largest gain, consistent with Google's July serving-stack update.
Kimi K2 Thinking and GPT-5.1 were stable within run-to-run variance (±0.4 points across three seeds).
No model regressed beyond variance bounds this cycle.

Full per-model numbers, with dates and configuration, are on the leaderboard.

Measurement notes

Runs used vendor-default reasoning settings, a 128K context ceiling for parity, and the agentless-lite scaffold documented in the methodology post. Three seeds per model; we report the median. Scores for models whose vendors publish higher numbers under custom scaffolds are flagged unverified on model pages until we can reproduce them.

One harness change this cycle: we upgraded the container image to match SWE-bench's July revision, which resolved two flaky test environments in the django subset. The change affected all models identically and moved no median by more than 0.2 points.

September's refresh is scheduled for the first week of the month; subscribe to the changelog via RSS to get refresh announcements as they land.

Kimi K2 Thinking adds native tool-call streaming in the Moonshot API

Sat, 01 Aug 2026 12:00:00 GMT

Moonshot AI enabled native tool-call streaming for Kimi K2 Thinking on August 1, 2026. Tool-call arguments now stream incrementally rather than arriving as completed blocks, cutting time-to-first-action in agent loops. The change is API-side, requires no model update, and matches equivalent features shipped by Anthropic and OpenAI.

Moonshot AI enabled incremental streaming of tool-call arguments for Kimi K2 Thinking on August 1, 2026, per the platform documentation. Previously, the API buffered each tool call until its argument JSON completed — a design that added seconds of dead time per step in long agent chains.

What changed

With streaming enabled, agent frameworks receive partial argument deltas as the model generates them, in a format compatible with the OpenAI streaming convention most frameworks already parse. For K2 Thinking specifically — a model whose signature behavior is chains of hundreds of sequential tool calls — the latency saving compounds per step.

The change is server-side. Model weights are unchanged; self-hosted deployments already had access to raw token streams and are unaffected.

Why it matters

Tool-call streaming was one of the last API-surface gaps between K2 Thinking and the closed-model APIs it competes with. Anthropic and OpenAI both ship fine-grained tool streaming as standard. With the gap closed, framework-level integrations no longer need Moonshot-specific buffering paths — one less reason for agent builders to default to closed APIs.

K2 Thinking remains, in our tracked measurements, the open-weight model with the longest stable tool-call chains. API pricing is unchanged at $0.60 input / $2.50 output per million tokens (verified August 1, 2026).

How we benchmark models on llm.blog

Sat, 01 Aug 2026 12:00:00 GMT

llm.blog measures models through a fixed public harness: vendor-default settings, three seeds per benchmark, median reported, every score dated. We re-measure monthly, flag unreproduced vendor claims as unverified, and publish configuration diffs whenever the harness changes. This page is the canonical methodology reference for every number on the leaderboard.

Every number on the leaderboard comes out of the process on this page. When this page and a leaderboard cell disagree, the cell's measurement date tells you which is newer; the methodology described here is versioned, and harness changes are logged in the changelog with dates.

The five rules

Our measurement policy compresses to five rules:

Vendor-default settings. We run each model the way a developer gets it from the API with no tuning: default temperature, default reasoning effort, vendor-recommended system prompt if the docs specify one. Benchmarks are meant to predict what you get, not what is achievable.
Fixed public scaffold. Every model runs the same agentless-lite scaffold for SWE-bench Verified and the official containers for Terminal-Bench. Scaffold code is pinned by commit hash and linked from each measurement cycle's notes.
Three seeds, median reported. Single-run benchmark scores are noise wearing a suit. We run three seeds and report the median; the spread is retained internally and cited when a movement is within variance.
Every score carries a date. Serving-stack revisions move scores without version-number changes. A score without a measurement date is not a fact; it is a rumor with decimals.
Unreproduced numbers are flagged. Where we display a vendor-published figure we have not reproduced — new model, new benchmark, harness gap — it renders with an explicit unverified flag until we can measure it ourselves.

Benchmark selection

We track six benchmarks. The selection favors measures that predict production behavior for API practitioners over leaderboard-culture staples:

Benchmark	What it predicts	Known weakness
SWE-bench Verified	Repository-scale coding agents	Python-heavy; scaffold-sensitive
Terminal-Bench 2.0	Operational/terminal competence	Young benchmark, small task set
GPQA Diamond	Expert reasoning ceiling	Saturating at the frontier
AIME 2025	Structured mathematical reasoning	Contamination pressure grows yearly
MMMU	Multimodal understanding	Weak proxy for document extraction
LiveCodeBench	Contamination-controlled coding	Contest style ≠ production code

Two absences are deliberate. We do not track MMLU or its descendants: at frontier accuracy the residual questions are disproportionately mislabeled or ambiguous, so movement measures label noise. We do not track head-to-head preference Elo (arena-style rankings): preference voting measures persuasiveness under casual prompts, which diverges from task competence in ways that reward stylistic tuning.

Contamination policy

Contamination is not a scandal; it is a baseline condition to engineer around. Our posture:

Prefer benchmarks with rolling post-cutoff problem sets (LiveCodeBench) or verified human curation (SWE-bench Verified).
Track year-stamped sets (AIME 2025) only while the newest models' training cutoffs plausibly precede the problems, and retire them after.
Watch for the contamination signature — a model dramatically outperforming its own capability profile on one aged benchmark — and annotate rather than delete when we suspect it.

Variance and what counts as movement

Run-to-run spread on agentic benchmarks is larger than most coverage admits: ±0.4 points is typical on SWE-bench Verified at three seeds, and Terminal-Bench 2.0 runs wider. Our reporting rule: a month-over-month delta inside the observed seed spread is described as "stable," not as movement. Deltas beyond spread get called movement and, where we can identify it, a cause — serving update, harness revision, or unexplained.

When the harness itself changes — container image updates, scaffold pin bumps — we re-run affected benchmarks for all models in the same cycle, never just the new arrivals, and note the change in the cycle notes. A harness change that moves any median more than 0.5 points triggers a full historical annotation.

What the leaderboard is not

The leaderboard ranks measured capability on six axes. It does not rank:

Value. Price-per-capability is workload-specific arithmetic; the comparison pages do that math per pairing, and the calculator on each one does it for your token mix.
Safety or alignment properties. We do not have the harness to measure them credibly, and we decline to launder vibes into a column.
Latency and throughput. Serving performance varies by region, tier, and hour; point-in-time measurements would mislead more than inform. We may add percentile-tracked latency in future with continuous measurement.

Reproducing our numbers

Scaffold commit hashes, container digests, seed values, and per-task result JSON for each cycle are retained. The dataset behind the leaderboard — scores, dates, sources, flags — is the same JSON that renders the site, and the llms.txt file points machine readers at the canonical URLs. Cite the leaderboard with its measurement date and the numbers will stay attributable even after later refreshes move them.

Corrections: corrections@llm.blog. A correction that survives verification lands in the next cycle notes with credit.

OpenAI halves GPT-5.1 cached-input pricing to $0.125 per million tokens

Tue, 28 Jul 2026 12:00:00 GMT

OpenAI reduced GPT-5.1 cached-input pricing from $0.25 to $0.125 per million tokens on July 28, 2026 — 10% of the $1.25 base input rate. Base input and output prices are unchanged. Workloads reusing long system prompts, tool definitions, or document contexts see the largest savings.

OpenAI cut GPT-5.1 cached-input pricing to $0.125 per million tokens on July 28, 2026, down from $0.25, per the pricing documentation. Cached input now bills at 10% of the $1.25 base input rate. Base input and output ($10 per million) prices are unchanged.

Who saves

Prompt caching applies to repeated prefixes: system prompts, tool schemas, few-shot blocks, and pinned document context. The workloads that benefit most:

Agents with large tool manifests. A 20K-token tool schema reused across a 50-step session now bills the repeated portion at $0.125 per million — an 87.5% discount against uncached input.
High-frequency RAG over stable corpora. Pipelines that pin document context across many queries convert most of their input volume to the cached rate.
Multi-tenant chat products with shared system prompts across users on the same cache shard.

Cache TTL and eligibility rules are unchanged; only the rate moved.

Competitive context

The cut narrows the effective-price gap with Claude Sonnet 4.5, whose cache-read pricing has been $0.30 per million tokens (10% of its $3 base) since launch. Both vendors now anchor cached input at 10% of base — the emerging industry convention — leaving the base rates as the real comparison. Current effective prices for common token mixes are in our GPT-5.1 vs Claude Sonnet 4.5 comparison, updated July 28, 2026.

Alibaba ships Qwen3-Max 2026-07 update with improved long-context recall

Wed, 22 Jul 2026 12:00:00 GMT

Alibaba released a Qwen3-Max update on July 22, 2026, improving retrieval accuracy in the second half of its 256K-token context window. The update is transparent to API callers — same model ID, same tiered pricing from $1.20 per million input tokens. Alibaba reports no changes to reasoning benchmark scores.

Alibaba deployed an updated Qwen3-Max on July 22, 2026, per the Qwen team blog. The update targets long-context retrieval: Alibaba reports materially improved needle-in-haystack and multi-hop recall between 128K and 256K tokens, the region where the model previously degraded fastest.

What changed

The update is a serving-side model revision under the same qwen3-max API identifier — callers get it automatically. Alibaba states reasoning and coding benchmark scores are unchanged, and our tracked scores for the model keep their existing measurement dates until the August refresh confirms.

Tiered pricing is unchanged, starting at $1.20 per million input tokens and $6 per million output for the first tier (verified July 22, 2026, Model Studio documentation).

Practitioner notes

Long-context claims are the least transferable numbers in vendor announcements: recall curves depend heavily on task shape, and vendor-reported needle tests overstate practical retrieval in structured-extraction workloads. We flag Qwen3-Max's long-context figures unverified on its model page until the September measurement cycle, which will add a long-context retrieval track to our methodology.

Teams already running Qwen3-Max on document-heavy workloads should A/B the update against archived outputs — same-ID serving revisions are exactly the case where silent behavior drift bites production pipelines.

Structured-output verification patterns for production LLM pipelines

Mon, 20 Jul 2026 12:00:00 GMT

Schema-constrained decoding eliminates malformed JSON but not wrong JSON: type-valid hallucinated values, unit drift, and cross-field contradictions survive it. Production pipelines need layered verification — schema, semantic invariants, provenance grounding, statistical monitoring, and selective human escalation. This piece documents each layer with implementation patterns and observed failure rates.

Every provider now ships schema-constrained decoding, and every provider's documentation quietly overpromises what it buys you. Constrained decoding guarantees the output parses. It does not guarantee the output is true, grounded, or internally consistent. The gap between those guarantees is where production incidents live.

This piece documents the verification stack we run on a document-extraction pipeline processing roughly 40K documents daily against frontier-model APIs, with the failure rates each layer catches. Numbers are from June–July 2026 production traffic; your distribution will differ — measure it.

The failure taxonomy schema validation can't see

Schema-valid failures cluster into four families:

Type-valid hallucination. The model returns "invoice_total": 4820.00 — a perfectly typed float that appears nowhere in the document. Constrained decoding increases the risk at the margin: when the model is uncertain, the grammar forbids it from expressing uncertainty outside the schema.
Unit and format drift. Dates parsed day-first from US documents, totals in cents where the schema assumes dollars, percentages as 0.15 vs 15. Type-correct, silently wrong by 100×.
Cross-field contradiction. Line items that don't sum to the stated total; an end date before a start date; a currency: EUR beside a $-prefixed source span.
Enum coercion. Forced to choose from ["invoice", "receipt", "credit_note"], the model files a purchase order as an invoice rather than failing. Closed enums without an escape value convert "I don't know" into confident misclassification.

In our pipeline, 2–4% of schema-valid extractions exhibit at least one of these per day, varying with document mix.

Layer 1: schema — but design for refusal

The schema layer is table stakes; the design detail that matters is giving the model somewhere to put uncertainty:

{
  "doc_type": { "enum": ["invoice", "receipt", "credit_note", "other"] },
  "confidence": { "enum": ["high", "medium", "low"] },
  "fields_not_found": { "type": "array", "items": { "type": "string" } }
}

Adding "other" plus a fields_not_found array cut our enum-coercion rate by more than half. The model was never unable to say "I don't know" — the schema had just made it grammatically impossible.

Nullable-with-reason beats required-with-guess for every field a document might legitimately lack.

Layer 2: semantic invariants

Every schema ships with unstated physics. Write them down as executable checks that run on every extraction:

def invariants(x: Extraction) -> list[Violation]:
    v = []
    if x.line_items and abs(sum(i.amount for i in x.line_items) - x.total) > 0.01:
        v.append(Violation("line_items_sum", severity="block"))
    if x.due_date and x.issue_date and x.due_date < x.issue_date:
        v.append(Violation("date_order", severity="block"))
    if x.total and not (0 < x.total < 10_000_000):
        v.append(Violation("total_range", severity="review"))
    return v

Invariants are cheap, deterministic, and catch the contradiction family almost completely. Ours run in three severities: block (re-extract with violation appended to the prompt), review (human queue), log (monitoring only). The re-extract-with-feedback loop resolves roughly 70% of block violations on the first retry; the remainder queue for review.

The prompt-side counterpart matters as much: state the invariants in the extraction prompt. Models violate constraints they were never told about at several times the rate of stated ones.

Layer 3: provenance grounding

Type-valid hallucination survives layers 1 and 2 when the invented value is plausible. The countermeasure is requiring the model to cite its work — every extracted value paired with a source span:

{ "total": 4820.00, "total_span": "TOTAL DUE: $4,820.00" }

Verification then checks the span actually occurs in the source (fuzzy match tolerating OCR noise and whitespace) and the value is derivable from it (number extraction on the span, comparison against the field). Span-not-found or value-mismatch routes to review.

Grounding catches the failures that matter most — invented financial values — at a cost: span fields roughly double output tokens for dense schemas. We ground only the six fields whose wrongness is expensive, not all forty. Failure rate at this layer: ~0.5% of extractions that passed layers 1–2, which is exactly the population you could not have found otherwise.

Layer 4: statistical monitoring

Per-document verification misses distribution-level drift: a serving-stack update that shifts date formats, a new document template that quietly halves field-found rates. Distribution monitors watch daily aggregates per field — null rate, mean, P95, enum mix — against trailing 28-day bands, alerting on excursions.

This is the layer that catches silent model updates. Same API identifier, new serving revision, moved behavior: we have caught three such shifts in a year of operation, each visible in field-level distributions days before any per-document check fired. If your pipeline pins vendor aliases rather than dated snapshots, this layer is not optional.

Layer 5: selective human escalation

Escalation is a budget allocation problem: review capacity is fixed, so route it where per-item expected loss is highest. Our routing score is a weighted sum of layer signals — invariant review flags, grounding mismatches, model-reported low confidence, and monetary value of the document. The result: ~1.5% of daily volume queues for human review, and that 1.5% contains an estimated 80%+ of surviving errors (estimated via periodic random-sample audits — which you must also run, or your escalation model grades its own homework).

The stack, summarized

Layer	Catches	Cost	Our observed catch rate
Schema + refusal design	Malformed output, forced guessing	~0	eliminates class
Semantic invariants	Contradictions, range violations	CPU-trivial	2–4% of valid outputs flagged
Provenance grounding	Plausible hallucination	~2× output tokens on grounded fields	~0.5% post-invariant
Distribution monitoring	Silent drift, template shifts	infra-light	3 serving shifts / year
Selective escalation	Residual tail	fixed human budget	~80% of survivors in 1.5% of volume

None of these layers is novel; the system property comes from running all five. Schema validation alone is a seatbelt bolted to a car with no brakes — it makes the crash tidier, not less likely.

What we'd build differently today

Two revisions from a year of operation. First, we would adopt grounded spans from day one rather than retrofitting them after the first hallucinated-total incident; retrofit cost exceeded first-build cost by an embarrassing multiple. Second, we would version extraction prompts and schemas in the same artifact with the same review gates as code — the worst drift we shipped came from a "harmless" prompt wording change that moved enum distributions 9%, discovered by layer 4 eleven days later. Everything upstream of the model call is code; treat it with code's discipline.

Terminal-Bench 2.0 joins the llm.blog tracked benchmark set

Wed, 15 Jul 2026 12:00:00 GMT

llm.blog added Terminal-Bench 2.0 to its tracked benchmark set on July 15, 2026, scoring six models in the first cycle. Claude Opus 4.5 leads at 59.3%. Terminal-Bench measures task completion in a live terminal environment — operational competence that SWE-bench's repository-patching format does not capture.

We added Terminal-Bench 2.0 to the tracked benchmark set on July 15, 2026, and scored six models in the first measurement cycle: Claude Opus 4.5 (59.3%), Gemini 3 Pro (54.2%), Claude Sonnet 4.5 (50.0%), Kimi K2 Thinking (47.1%), Claude Haiku 4.5 (41.0%), and Grok 4.1 (pending a harness fix). Remaining models join in August. All scores and dates are on the leaderboard.

Why Terminal-Bench

Terminal-Bench drops a model into a live terminal and scores task completion: environment setup, debugging, data wrangling, service configuration. It measures operational competence — the "can it actually drive a computer" question — where SWE-bench Verified measures repository-scale code modification. The two disagree in informative ways: models tuned aggressively for patch generation can underperform on open-ended terminal tasks, and the gap predicts real-world agent reliability better than either number alone.

Configuration

We run the standard 2.0 task set in the official container, vendor-default reasoning settings, 20-minute per-task timeout, two attempts with the better run discarded (we report first-attempt scores; the second run measures variance only). Full harness details are in the methodology, which has been updated with the Terminal-Bench section as of July 15, 2026.

Vendor-published Terminal-Bench numbers exist for several tracked models and differ from ours by up to four points, mostly attributable to scaffold differences. Model pages flag vendor figures unverified where we have not reproduced them.

DeepSeek extends off-peak API discount window to 12 hours daily

Wed, 08 Jul 2026 12:00:00 GMT

DeepSeek extended its off-peak API discount window to 12 hours daily on July 8, 2026, up from 8. Requests between 16:30 and 04:30 UTC bill at roughly 50% of standard rates — about $0.14 input and $0.21 output per million tokens for DeepSeek-V3.2. Batch workloads gain the most.

DeepSeek widened its off-peak discount window to 12 hours daily on July 8, 2026, per the API pricing page. Requests submitted between 16:30 and 04:30 UTC now bill at approximately half of standard rates — for DeepSeek-V3.2, roughly $0.14 per million input tokens and $0.21 per million output, against the standard $0.28 / $0.42.

The arbitrage

Off-peak pricing rewards workloads that can shift: nightly embedding refreshes, evaluation sweeps, synthetic-data generation, batch summarization. A pipeline moving 5B tokens of monthly batch volume into the window saves about $700 per month at V3.2 rates — on prices that were already the lowest in our tracked set.

The mechanics are simple: pricing is determined by request submission time, no reservation or commitment required. That makes the window one config change for most batch schedulers.

The bigger pattern

Time-of-day pricing is DeepSeek exporting its infrastructure reality: its GPU fleet's demand curve follows Chinese business hours, and the discount window sells the trough. No Western lab has followed — OpenAI, Anthropic, and Google price batch workloads through dedicated batch APIs with 50% discounts and delayed completion instead. The two mechanisms converge on similar effective rates; DeepSeek's version returns results synchronously, which matters for evaluation loops that gate on results.

Standard-rate comparisons across all tracked models are on the leaderboard; the DeepSeek-V3.2 page carries the full pricing history.

Claude Sonnet 4.5's 1M-token context window graduates from beta

Thu, 02 Jul 2026 12:00:00 GMT

Anthropic moved Claude Sonnet 4.5's 1M-token context window from beta to general availability on July 2, 2026. The beta header requirement is removed. Prompts above 200K tokens bill at premium rates — $6 input and $22.50 output per million tokens — doubling effective input cost for full-window workloads.

Anthropic promoted Claude Sonnet 4.5's 1M-token context window to general availability on July 2, 2026, per the context windows documentation. The context-1m beta header is no longer required; all API traffic can address the full window.

The pricing structure

The window is tiered. Prompts up to 200K tokens bill at the standard $3 input / $15 output per million. Prompts above 200K bill the entire request at the long-context rate of $6 input / $22.50 output per million tokens (rates verified July 2, 2026).

That cliff shape matters for pipeline design: a 210K-token prompt costs more than twice a 195K-token prompt on input. Retrieval pipelines that can compress below the boundary should; the premium tier is priced for workloads that genuinely need single-pass reasoning over massive context — codebase-scale analysis, long-document synthesis, extended agent sessions with full history.

Competitive position

GA closes most of the context gap with Gemini 3 Pro, whose 1M-token window has been standard since its November 2025 launch with a gentler tier boundary ($4 input above 200K vs Anthropic's $6). For pure long-document work, Gemini remains cheaper at the top of the window; Sonnet's case rests on its agentic-coding lead. Both patterns are priced out in our head-to-head.

The Claude Sonnet 4.5 model page reflects GA status in its spec table as of July 2, 2026.