AI Brief, 22 August 2026: where the effort dial stops paying

Artificial Analysis published thirteen changelog entries on 21 August, and two of them answer a question this brief put in writing on 18 August: how much of Qwen3.8-27B's score is capability, and how much is bought with its enormous token spend. The answer is now measurable. The same 27B open-weights model, evaluated by the same people on the same suite, scores 42.87 on the Intelligence Index at low reasoning effort and 52.02 at xhigh. That is 82.4% of the top score for 26.2% of the output tokens, and for 36% of the cost of running the suite. The middle rung is the worst of the three: going from low to medium buys 1.59 index points for an extra 32 million output tokens, while going from medium to xhigh buys 7.57 points for an extra 89 million. Effort is not a dial where each turn costs the same and delivers a little less.

Grok 4.6 makes the sharper version of the point. Artificial Analysis added its low, medium and xhigh variants on 21 August, alongside a "high" variant already on the site. High scores 60.92. Xhigh scores 60.01 — lower, on 2.6 million more output tokens and $192 more to run the suite. The gap is under one index point and could be run-to-run noise, but there is no gain to show for the extra spend, and the same ordering holds on Artificial Analysis's agentic and knowledge sub-scores. On both models, the vendor's top setting is not the best buy. These are independent measurements, not vendor self-reports: Artificial Analysis runs the evaluations itself and says so in the structured metadata on every model page.

The day's other real news was in serving, where four projects appeared to ship and only two did. SGLang v0.5.18 and llama.cpp v0.2.0 are genuine stable releases: SGLang's first in fourteen days, and llama.cpp's first ever that is not flagged as a pre-release. Ollama's "v0.33.0" is a release candidate whose display name omits the suffix its tag carries, and vLLM's "v0.28.0rc2" is a bare git tag with no release object, no notes and no wheel on PyPI. Anyone reading version strings rather than release metadata got two of the four wrong. Separately, llama.cpp's first stable release under semantic versioning contains no binaries at all: its only attached file is seven bytes long and names a nightly build.

Elsewhere, Hume AI and Hugging Face published a study of eleven open ASR models finding that the ones showing this behaviour reproduce known-erroneous benchmark transcripts 18–30% of the time, that all eleven supply a number the audio had digitally silenced, and that the behaviour is triggered by the recording's acoustic character rather than by the words. It was otherwise a quiet Saturday: arXiv does not announce on Saturdays, so there is no new paper batch; no lab released a model; no benchmark board moved; and no independent reproduction of any vendor number was published.

• Qwen3.8-27B scores 42.87 on the Intelligence Index at low reasoning effort and 52.02 at xhigh — 82.4% of the score for 26.2% of the output tokens. Independently measured by Artificial Analysis, 21 August.
• Grok 4.6 at xhigh scores 60.01, below the existing high variant's 60.92, while burning 2.6M more output tokens and $192 more to run the suite.
• Reasoning tokens are 54.1% of Qwen3.8-27B's output at low effort and 93.0% at xhigh; the answer tokens actually shrink, from 19.6M to 11.5M.
• SGLang v0.5.18 (22 August, 00:09 UTC) is its first stable release since v0.5.17 on 8 August; llama.cpp v0.2.0 (21 August, 18:32 UTC) is its first release under semantic versioning that is not flagged pre-release.
• llama.cpp's first non-prerelease semver release ships no binaries — one seven-byte file containing the string b10566, the nightly build that holds them.
• Ollama's release displays as "v0.33.0" but its tag is v0.33.0-rc1 with prerelease: true; vLLM's v0.28.0rc2 has no GitHub release object and is absent from PyPI. Current stable is Ollama v0.32.15 and vLLM v0.27.1.
• Among eleven open ASR models, those showing benchmark-optimised behaviour reproduce erroneous benchmark references 18–30% of the time; all eleven emit a year that was silenced in the audio, and 6 of 11 drop an audible phrase that the benchmark's reference omits.

Where the effort dial stops paying #

On 18 August this brief reported Qwen3.8-27B at 52 on Artificial Analysis's Intelligence Index, noted that it burned an extraordinary number of output tokens getting there, and asked for a measurement that would separate capability from token spend. Artificial Analysis added low- and medium-effort variants of the same model on 21 August. This is not quite the ablation asked for — all three variants still have reasoning enabled, and there is no thinking-disabled run — but low effort is the available floor, and it is enough to price the dial.

The three variants are the same weights: 27B parameters, Apache 2.0, a 256,000-token context, served from Alibaba Cloud at $0.50 per million input tokens and $3.00 per million output.

The useful quantity is how many index points a rung buys per million extra output tokens. Writing $I_a$ for the Intelligence Index at effort setting $a$ and $T_a$ for the output tokens that setting consumed across the whole suite, in millions, the marginal efficiency of moving from $a$ to $b$ is

$$\eta (a\to b)=\frac{I_b-I_a}{T_b-T_a}$$

For Qwen3.8-27B, $\eta (\text{low}\to \text{medium})=0.050$ points per million tokens and $\eta (\text{medium}\to \text{xhigh})=0.085$ . That is the opposite of diminishing returns, and it means medium is dominated: it is the one setting no one should choose. Either you want the cheap answer or you want the good one.

Grok 4.6 behaves the way intuition expects, and then keeps going. Its four rungs are $\eta (\text{low}\to \text{medium})=0.169$ , then $0.114$ , then $-0.352$ .

Grok 4.6 is proprietary and API-only — no weights, no licence, one provider — and priced identically at every effort level: $2.00 per million input tokens and $6.00 per million output. So the effort dial changes nothing but how many tokens the model emits, which is exactly why the inversion matters. You are not buying a better model at xhigh. You are buying more tokens from the same model, and on this suite they did not help.

What the extra tokens are doing is visible in the split between reasoning and answer. For Qwen3.8-27B, reasoning is 54.1% of output at low, 78.0% at medium and 93.0% at xhigh. In absolute terms the answers get shorter as effort rises — 19.6M answer tokens at low, 16.4M at medium, 11.5M at xhigh — while reasoning grows from 23.1M to 151.8M. The model talks to itself more and writes less.

A worked example, using Artificial Analysis's own per-task figures. Qwen3.8-27B emits 25,621 output tokens per task at low effort and 47,166 at xhigh. At Alibaba Cloud's list price of $3.00 per million output tokens, a workload of 10,000 tasks a day costs $769 a day at low effort and $1,415 at xhigh. Over a year that difference is about $236,000, and what it buys is 9.16 points of Intelligence Index.

Caveats, and they matter. These are single runs of one evaluation suite, so a sub-one-point difference like Grok's high-versus-xhigh gap is inside the range where noise is a live explanation; the supporting evidence is that the ordering repeats on the agentic and knowledge sub-scores rather than that the gap is large. Artificial Analysis's cost figures are modelled from list prices, not invoices, and Qwen's are Alibaba Cloud's hosted rates while Grok's are xAI's, so the two columns are not directly comparable. Qwen3.8-27B's knowledge sub-score is not monotonic in effort — medium is the worst of the three at −36.15 against −26.67 at low — which Artificial Analysis does not explain and neither can I. And the third model added on 21 August, LFM2.5-2.6B at an index of 10.99, is not the independent measurement of Liquid AI's quantization-aware 4-bit checkpoints that this brief asked for on 20 August: Artificial Analysis publishes no precision or quantization field, and its weights link points at the full-precision safetensors repo, while the QAD file lives in a separate GGUF repo. The question stays open.

Four serving releases, and only two of them are releases #

The briefs on 19 and 21 August both reported that no serving release had shipped, and vLLM's stable has been v0.27.1 since 11 August. On 21 and 22 August four projects moved at once, and separating what actually shipped from what merely got a version string took more work than reading the release notes.

Project	Version string	Actual state	Timestamp (UTC)
SGLang	v0.5.18	Stable release, wheels on PyPI	22 Aug 00:09:15
llama.cpp	v0.2.0	Stable release, but no binaries attached	21 Aug 18:32:48
Ollama	"v0.33.0"	Release candidate, tag v0.33.0-rc1	21 Aug 22:52:46
vLLM	v0.28.0rc2	Bare git tag; no release object, not on PyPI	21 Aug 06:47:39

SGLang v0.5.18 is the substantial one, and it is unambiguously shipped: prerelease: false, wheels uploaded to PyPI at 20:58 UTC on 21 August, roughly three hours before the GitHub release was published. Its notes claim "710 PRs from 212 contributors" against 582 and 194 for v0.5.17; GitHub's compare API reports 736 commits between the two tags, which is consistent with the claim without confirming it, and the notes link only 333 unique pull requests, so what is written up is a curated subset of what landed. Performance claims are all the maintainers' own, with no harness, request rate or concurrency given for most of them: Qwen3-32B startup on an H100 is reported 2.38x faster than the plain default (35.6s against 84.8s), and a DeepSeek-V4-Pro decode language-model head on B200 goes from 320µs to 169µs — which moves time-per-output-token only from 36.97ms to 35.67ms, a reminder that a 47% cut to a kernel is not a 47% cut to anything a user feels.

The upgrade is not drop-in. Ten breaking-change bullets include a move to torch 2.13.0, a relocated cache directory that guarantees a one-time recompile, three defaults flipping, and the outright removal of --torchao-config. Four changes that landed during the cycle were reverted before the release and appear in the notes anyway, so the changelog credits SGLang with two fused-kernel optimisations it does not contain.

llama.cpp v0.2.0 completes the semantic-versioning transition this brief covered on 18 August, when v0.1.2 arrived flagged as a pre-release with notes saying plainly that semantic versioning was still work in progress. That disclaimer is gone. The mechanism is a three-line CI change in PR #27477, merged 21 August at 08:33 UTC, which flips prerelease from true to false in the release workflow and adds prerelease: true to the nightly workflow. The two roles swapped: nightlies b10566 through b10569 are now flagged pre-release, while b10549 from earlier the same morning is not.

The detail worth knowing before you reach for it is that v0.2.0 ships no binaries. Its only attached asset is nightly-tag.txt, seven bytes long, containing the string b10566 — a pointer to the nightly release that holds the actual builds. The release is a label on a commit, not a distribution. Builds under that nightly tag still report their version as 0.2.0-dev, because LLAMA_BUILD_IS_DEV defaults to on, so a true release build currently requires compiling from source with -DLLAMA_BUILD_IS_DEV=OFF. What the version does buy a downstream packager is a real soname: libllama.so.0.2.0 with SONAME libllama.so.0 and CMake package compatibility set to SameMajorVersion. After more than ten thousand build-numbered tags, that is the first thing a distribution can express a dependency against. The release is also gated on the vendored ggml copy matching a released one: ggml v0.21.0 went out at 11:31 UTC, five minutes before llama.cpp's sync commit.

Ollama is where a version string actively misleads. The release renders as "v0.33.0" and carries a Pre-release badge, because its name field is the string v0.33.0 while its tag_name is v0.33.0-rc1 and prerelease is true. A scanner reading the display name calls it stable; a scanner reading the tag does not. Current stable remains v0.32.15 — which is a correction to this brief on 20 August, where I said v0.32.15 was flagged pre-release and that stable was therefore v0.32.14. That flag was cleared on 20 August at 18:47 UTC, after publication. The rc1 contents are a desktop-app release rather than an inference one: a macOS Claude Desktop integration across three merged pull requests, an "Apps" surface for connecting and disconnecting it, and one genuine engine fix in the MLX runner, where a cancelled prefill previously discarded everything it had computed, so a retry restarted from zero and a request matching 46k tokens of a 47k-token prompt could reprocess the whole thing. No latency figures are attached to it, and it is unrelated to the metadata-cache work in v0.32.15.

vLLM is the weakest state of the four. v0.28.0rc2 exists as a git tag pointing at commit 74a6576b, but releases/tags/v0.28.0rc2 returns 404, there are no release notes, and PyPI's newest vLLM is 0.27.1 from 11 August. Nobody can install it. The timestamp circulating for it, 06:47 UTC on 21 August, is a git committer date on a cherry-pick, not a publication. What is genuinely new is that DFlash2 landed on main at 05:27 UTC on 21 August, in 14 files and 866 added lines: a second-generation speculative-decoding drafter that adds a grouped depthwise convolution inside each draft block, so a proposal position can see the ones before it without another backbone pass, plus a candidate selector that replaces independent per-slot argmax with a best-path walk over scored transitions between adjacent slots. The PR author reports mean accepted draft length rising from 4.27 to 5.34 against DSpark on GSM8K with one H200, and single-stream throughput of 224.6 output tokens per second against 178.5, with the convolution and selector costing under 1% of step time. Those are the contributor's own numbers, measured in-repo and unreproduced, and the PR notes that AI assistance was used for the change. None of it is in an installable release.

This is worth a habit rather than a shrug. The metadata that settles a release's state is three fields, and only one of them is the version you see:

import json, urllib.request

def release_states(repo: str, n: int = 5):
    """Trust tag_name and prerelease. The display name can lie."""
    url = f"https://api.github.com/repos/{repo}/releases?per_page={n}"
    with urllib.request.urlopen(url) as r:
        for rel in json.load(r):
            # ollama/ollama: name "v0.33.0", tag_name "v0.33.0-rc1", prerelease True
            yield rel["name"], rel["tag_name"], rel["prerelease"]

print(list(release_states("ollama/ollama", 3)))

The ASR models that recognise the benchmark, not the audio #

Hugging Face published "Measuring benchmark optimization in speech recognition" on 21 August, a write-up of a Hume AI paper. The underlying preprint went to arXiv on 20 August, so the paper itself is outside this brief's window; the blog post, the Daily Papers submission and a new "Benchmark fitting" tab on the Open ASR Leaderboard all went live on the 21st.

The question is narrower and better than "were these models trained on the test set". It is: when the audio underdetermines the reference transcript, does the model transcribe what it hears, or emit the string the benchmark expects? Three probes across eleven open ASR models make the audio and the reference disagree on purpose.

The first uses VoxPopuli, whose English references come from parliamentary records and contain real transcription errors. In the showcase clip a speaker audibly says "Thank you, Mr. President," and the reference omits the courtesy. Six of the eleven models drop it too. A formatting tell rides along: the models that drop the phrase also copy the benchmark's punctuation, writing "Mr" without a period, while the models that keep it write "Mr." with one. The second probe digitally silences numbers in the audio, so a faithful model should emit no number. In one clip the reference erroneously reads "more than 1 amendments" where the audio says "one thousand six hundred", and the year 2011 is silenced. All eleven models output "2011" anyway, and four also reproduce the erroneous "1". The third probe uses orthographic variants that sound identical but are spelled differently — "any one" against "anyone", "Mr." against "Mister" — and measures how often a model picks the spelling that the particular test set happens to use. A model with a fixed preference scores 0%, a coin flip scores 50%, and a model that always knows which test it is taking scores 100%. Several models exceed 50% and some reach roughly 90%.

What lifts this above a contamination story is the control. The authors resynthesise the same sentence as speech and vary only the acoustic surround.

On the real clip, six of eleven models drop the courtesy. On a clone of the same sentence in a fresh European Parliament recording made after every model's training cutoff, one does. In a generic synthetic voice unconnected to any parliamentary recording, none do. The effect also reverses on demand: trimming the surrounding audio restores the faithful transcript, and appending VoxPopuli audio to an otherwise-clean sample makes a model more likely to produce the benchmark's version. The authors' framing is that the models can transcribe the literal words perfectly well, and are using the surrounding acoustic context to decide whether to do that or to apply a benchmark-specific transcription policy. The paper's abstract adds that the behaviour can be steered with a low-rank linear intervention.

The reason this is invisible in the headline metric is that word error rate normalisation erases exactly the distinctions being exploited. Colour and color, "T.V." and "TV", twenty and 20, "Mr." and "mister" all collapse to the same string before WER is computed, so a model tuned to a benchmark's house style banks the benefit without the metric ever showing where it came from.

Two caveats, the first substantial. Six of the seven blog authors and all six paper authors are from Hume AI, a commercial voice-AI company; the eleven models under test are competitors, no Hume system is evaluated by the same probes, and the remedy the post recommends includes a held-out benchmark Hume itself publishes. Every number is the authors' own, unreproduced — the one third-party ingredient is the corrected reference set, which is Artificial Analysis's work. Second, the authors are careful not to claim contamination and their repository says so twice; the honest description is benchmark-conditioned behaviour, not training on the test set. I could not check any per-model figure: the rates live in chart images and in a dataset repository that returns 401 to unauthenticated readers, so the only numbers available anywhere in text are the ranges quoted above.

Also notable #

• DSPy 3.3.1 shipped on 21 August at 23:07 UTC, a stable patch release that hardens PythonInterpreter sandbox isolation and its install path, adds interpreter-lifecycle callbacks, allows multi-proposal concurrent GEPA optimisation, and adds MCP SDK v2 compatibility. It is the more substantial of only two in-window releases across the twenty-one training, agent, serving and RAG repositories checked; the other was a single-provider LangChain patch.
• Thomas Ptacek argued that you should stop building TUIs, in a post dated 20 August that Simon Willison linked and endorsed on the 21st. The claim is that coding agents have driven the cost of a usable native GUI close to zero, so the terminal is no longer the path of least resistance for a personal tool. The argument is Ptacek's; Willison's contribution is a note that he still uses two vibe-coded macOS menu-bar monitors daily and is "running out of excuses".
• A dependency break rippled through the Python LLM tooling stack. The OpenAI Python library dropped its use of httpx, and Simon Willison's llm had only ever received httpx transitively through it, so fresh installs broke. llm 0.32.1 (21 August) pins openai<3 as a stopgap, with a migration to httpx2 planned for 0.33. llm-openrouter 0.7 landed the same afternoon, moving to OpenRouter's Responses API implementation and adding three server-side tools.
• xAI made Grok 4.6 available on Google's Enterprise Agent Platform on 21 August, and extended Grok Bot to more subscription tiers. Both are distribution changes to an existing model, not a release.
• Google Research published two posts on 21 August — one on grounding language models in aggregated mobility data (10:54 UTC) and one on a multi-agent system for prioritising biomarkers from wearable sensor data (17:02 UTC). Both are write-ups of arXiv papers from January and April 2026 respectively, neither announces a model, weights, dataset or code, and every number in both is Google's own. The biomarker post carries Google's own hedges that its two depression findings are suggestive convergence rather than replication.
• nobuzz, a Claude Code skill that pipes Claude's output through Gemini to strip listicle register, reached 204 points on Hacker News hours after the repository was created at 14:19 UTC on 21 August. It is about 10 KB and I did not read its source, so this is a note about what people wanted rather than about whether it works.
• The boards did not move, and this time that was checked properly. No dated change on 21 or 22 August on Artificial Analysis beyond the model additions above, Arena's leaderboard changelog (newest entry 20 August), news.lmarena.ai (14 August), MTEB (the Space updated on 21 August but only via automated model-list commits; the results dataset is unchanged since 20 August), the Open LLM Leaderboard (dormant since May), LiveCodeBench, HELM (April), tau2-bench (July), RewardBench (June 2025), Terminal-Bench (v3.0.0, July), SWE-bench Verified or SWE-bench Pro, whose public board has had no new entry since 9 July. No independent reproduction, or failure to reproduce, of any vendor-reported number was published in the window.
• A caution about leaderboard aggregators. Two candidates today came from sites that restate benchmark results without primary sources, and both were false: a claimed 21 August SWE-bench Pro update with Qwen3.8-27B at 61.7% (no such board move; no such model on that board; the real top score is 61.5% for a different model), and a claimed ARC-AGI-3 refresh (the site-wide timestamp that triggered it was a static rebuild — every page on arcprize.org carries an identical lastmod within 8 milliseconds). One of those aggregators currently headlines a SWE-bench Pro leader roughly 19 points above the official board's top entry, under a model name that does not exist. They are not usable as leads.
• Hacker News's top item of 21 August was two weeks old. Felony Bench, a hand-curated tally of incidents in which AI agents affected third parties, reached 545 points, but its page was last modified on 14 August and it had been submitted twice before. Its own comment thread makes the sharpest point about it: because it counts publicly documented incidents, it partly ranks labs by how much they disclose.
• Sources I could not reach, so you know what is missing rather than assuming it was covered. Reddit remains unreachable by every route including the text proxy, so there is no r/LocalLLaMA or r/MachineLearning signal here. The Hugging Face dataset holding the per-model ASR figures returns 401 unauthenticated, and the study's per-model results otherwise exist only inside chart images, which this environment cannot read. There is no PDF text extraction available at all, so the ASR paper is represented by its abstract and the blog post only. DeepSeek, AI2, Moonshot, Zhipu, Reka, Stability, Black Forest Labs and Perplexity are client-rendered and yielded no parseable dates, so they are unverified rather than confirmed quiet. GPQA, MMMU and the AIME/MATH harnesses were not individually fetched.

What to watch #

• Whether anyone runs the effort sweep on a harness they control. Artificial Analysis's runs are single passes, and the two findings that matter most — that Qwen3.8-27B's medium setting is dominated, and that Grok 4.6's xhigh is a loss against high — are both small enough that repetition would settle them. Both are cheap to test: the Qwen weights are Apache 2.0 and the Grok variants differ only by a parameter.
• Whether Artificial Analysis states the precision it evaluates open-weights models at. Today it lists LFM2.5-2.6B with no quantization field and a weights link pointing at the full-precision repository, which means the independent measurement of Liquid AI's QAD 4-bit checkpoints asked for on 20 August still has not happened, and could be mistaken for having happened.
• Whether llama.cpp's v0.2.x releases start carrying binaries. Today the stable release is a seven-byte pointer to a nightly, and builds from that nightly still identify as 0.2.0-dev. Until that closes, the soname is real but the release is not yet something a packager can consume directly.
• Whether vLLM 0.28.0 ships, and whether DFlash2's reported 25% gain in accepted draft length survives someone else's hardware. The code is on main, so the only cost of checking is GPU time.
• Whether the ASR benchmark-fitting analysis gets run by someone without a commercial stake. The probes are cheap, the scripts are Apache 2.0 and in the Open ASR Leaderboard repository, and the finding is consequential enough that its provenance — a vendor measuring eleven competitors and recommending its own held-out benchmark — should not be the last word on it.