AI Brief, 21 August 2026: the signals we score agents with

Four things published on 20 August, three of them papers announced in this morning's arXiv batch, land on the same joint: the signals used to train, score and monitor agents are measuring something other than what the people using them believe. Haiyue Zhang's audit of step-level credit assignment is the sharpest. It re-executes an agent's trajectory to establish, by replay rather than by annotation, what each action actually caused, and then asks how well two families of credit signal recover that. A large LLM judge and an implicit log-probability-derived signal both fail to separate from their own shuffled controls. The implicit signal correlates with how fluent the action was at 0.752, and with the action's measured causal effect at −0.004 once fluency is conditioned out.

Second, Microsoft released Thinkingbox, a sandbox and 507-task benchmark for agents in stateful business workflows, run at twenty independent attempts per task. The headline gap is between finding a solution and finding it every time: GPT-5.4 succeeds at least once in twenty attempts on 91.1% of tasks, and on all twenty attempts on 25.3%. The more useful number is buried in the appendix. Of the failed trials, 80.9% terminate cleanly and make a state-changing tool call, and 67.2% additionally end with no error in the final tool response, while 99.0% leave the database in the wrong state. The signals most production monitoring actually watches cannot see these failures.

Third, Artificial Analysis shipped reward-hacking detection into its live Coding Agent Index on 20 August, scoring flagged trials zero rather than excluding them, and moved the published rank order at the top of the board. It disclosed no count of flagged trials and no list of what was rescored. Separately, a benchmark released the same day measures specification gaming at up to 47.9% of runs in its worst configuration, using a detector whose false-positive rate its authors state plainly they did not estimate.

Those four are the day's story, and it is the second running about the measurement layer rather than about capability, after yesterday's compute-normalised finding that reward models correlate at roughly 0.12 with true quality. Not one of them is a capability result. They are evidence that the instruments are the weak part.

Elsewhere it was a quiet day. No frontier lab released a model, no major leaderboard published an independent reproduction, and the two inference releases worth reading both carry headline multipliers measured in their most favourable configuration: FlashPrefill V2's 47× is an attention-kernel microbenchmark that becomes 4.8× end-to-end and 1.2× with production chunked prefill, and Liquid AI's 3.2× speculative-decoding figure is a single batch-size-1 cell on an H100 whose cross-benchmark mean is 2.54×, falling to 1.18× on a MacBook.

• Zhang's replay audit: implicit credit correlates at 0.0193 with measured causal effect and an LLM judge at 0.1142, both overlapping their own marginal-matched shuffled controls; implicit credit correlates with action fluency at +0.752.
• Only 30.5% of the audited agent's turns are pivotal at all, meaning the outcome would have differed under an alternative the policy would actually have taken.
• Thinkingbox, 507 tasks × 20 trials × 14 models: GPT-5.4 scores 65.4% pass@1, 91.1% pass@20, 25.3% pass^20; about two-thirds of tasks are neither always solved nor never solved.
• Artificial Analysis Coding Agent Index v1.4, 20 August: reward-hacked trials now score 0, Terminal-Bench moved to v2.1, and reasoning tokens were split out of output tokens, changing published costs by up to a factor of four.
• FlashPrefill V2 claims 47.26× over FlashAttention-2. That is an attention operator microbenchmark; the same paper's end-to-end figure is 4.83×, and 1.21× once chunked prefill is switched on.
• Liquid AI released speculative-decoding drafters for three LFM2.5 models, headlined at 3.2×. That is one batch-size-1 cell on an H100; the cross-benchmark mean is 2.54× and the same model on a MacBook averages 1.18×.
• No serving release shipped. vLLM's stable is still v0.27.1 from 11 August, SGLang's v0.5.17 from 8 August; llama.cpp emitted only nightly build tags alongside the merged DSpark support.

What step-level credit actually measures #

Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay (arXiv 2608.19760, submitted 20 August 08:04 UTC, single author Haiyue Zhang) is a pre-registered study with frozen analysis plans and prompts in the appendices. That matters here more than usual, because the result is a null one and pre-registration is what separates a null from a fishing expedition that came up empty.

The problem it attacks is that step-level credit has no ground truth. If an agent takes twelve actions and the episode succeeds, which action deserves the credit? Every method in use answers by asking a model: an LLM judge scores each step, or credit is derived implicitly from the policy's own probabilities. Nobody checks those answers against what the steps actually did, because checking seems to require expensive human annotation of counterfactuals.

Zhang's instrument sidesteps annotation entirely by re-executing the environment. At each action turn $t$ , the factual action is replayed to a terminal outcome at least three times, never reusing the original continuation. Then $K=4$ distinct alternative actions are sampled from the same policy snapshot at collection temperature, and each is rolled to terminal at least three times. The causal effect of the action taken is the difference in mean outcome:

$$A_{\text{replay}}(t)=\mathbb{E}[o\mid \text{factual replays}]-\mathbb{E}[o\mid \text{alternative rollouts}]$$

where $o$ is the episode's binary success outcome. Nothing in that construction consults the credit signals being audited, which is the whole point: it is an instrument built independently of the thing it measures.

The environment is ALFWorld, a replayable text household-task setting with a binary outcome, and determinism was verified: zero divergences across 20,538 replay rollouts, re-checked on four independent hosts. The primary policy is Qwen2.5-7B-Instruct over 50 trajectories, yielding 1,768 complete turns, with Llama-3.1-8B-Instruct as a cross-family replication.

Two findings come out of it. The first is about the environment rather than the credit signals, and it is arguably the more durable: only 30.5% of complete turns are pivotal, 95% interval [28.4, 32.7]. At the other 69.5% the outcome is the same whatever the policy would plausibly have done. Any credit method is being asked to distribute credit across a trajectory where most steps did not matter.

The second is the audit itself. Within-trajectory Spearman correlation between credit and measured causal effect, median across trajectories:

Credit family	Correlation [95% CI]	Its own shuffled control
Implicit (log-probability derived)	0.0193 [−0.109, 0.081]	[0.005, 0.114]
LLM judge (Qwen2.5-72B-Instruct)	0.1142 [0.027, 0.168]	[−0.049, 0.117]

Both intervals overlap the interval produced by shuffling the same credit values against the same turns, which is the pre-registered test for a signal carrying no trajectory-specific information. Worth being precise about one thing the summaries going around get wrong: the judge is not at chance on every metric. On per-step sign agreement it scores 60.4% [52.1, 68.2], an interval that excludes 50%, so it does have some ability to say whether a step helped or hurt. What it does not have is the ability to rank steps within a trajectory, which is what credit assignment actually needs.

The mechanism is the part worth carrying away. The paper's one pre-registered positive prediction was that implicit credit tracks fluency, and it does: Spearman +0.752 [0.647, 0.793] against the action's mean token log-probability under the policy, $n=47$ trajectories, Holm-adjusted $p=0.0002$ , replicating at +0.701 on Llama. Regressing implicit credit jointly on fluency and on the replay-measured causal increment gives standardised weights of 0.955 and 0.402 respectively. Condition fluency out, and the partial correlation between credit and causal effect is −0.004, $p=0.87$ .

The scope limits are real and the paper does not hide them. One environment, one judge model at one size, 50 trajectories, binary outcomes. There is no frontier judge in the audit, so this is not evidence about GPT-5.6 or Opus 5 as a step scorer. A confidence-based router, the one mitigation tested, recovers 11.9% [9.4, 14.9] of pivotal turns. The artifacts are described as released through an anonymised repository for review and no URL is given anywhere in the paper, so nothing here is reproducible today. The licence is arXiv's default non-exclusive one, so the figures cannot be reused; the diagram above is drawn from the method description.

One success is not reliability #

One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv 2608.19741, submitted 20 August 07:37 UTC, twelve authors, code under microsoft/thinkingbox and microsoft/thinkingbox-data, CC BY 4.0) is a benchmark paper with no proposed method, which is unusual enough to say up front: it fixes nothing.

507 tasks across retail, travel, auto insurance, neobank IT and consulting, each run 20 independent times per model at temperature 1.0, for 10,140 trials per model across 14 models. Grading is executable: the terminal database state and side effects are checked, and wrong, missing or extra effects fail. 477 of the 507 tasks are graded on state alone.

The paper's two metrics are defined explicitly, and getting them the right way round matters. pass@k is the probability that at least one of $k$ attempts succeeds; pass^k is the probability that all $k$ succeed. So pass@20 is necessarily at least pass@1, and it is pass^20 that collapses.

The naive reading of that chart is that agents are chaotic. The arithmetic says the opposite, and this is the most interesting thing in the paper. If a model succeeded independently at 65.36% on each attempt, then all twenty succeeding would happen at ${0.6536}^{20}=0.02\mathrm{\%}$ . The observed 25.25% is roughly 1,250 times higher. For Claude Sonnet 4.6 the ratio is about 9,300×; for GPT-5.2, about 430,000×.

That is strong evidence of positive correlation between attempts. The model is not rolling a fresh die each time. It is close to deterministic per task, with per-task difficulty doing the work. The paper's own counts confirm it: for GPT-5.4, 45 tasks never pass in twenty attempts and 128 always pass, leaving 334 tasks, 65.9% of the suite, in an inconsistent middle. For Claude Sonnet 4.6 it is 58 never, 102 always, 347 inconsistent. Only about 9% of the benchmark is genuinely beyond the leading model. The rest is a coin-flip zone, and that is a different engineering problem from incapacity.

The finding I would actually act on is the evaluator ablation, over 79,853 failed trials. Among failed trials, 84.9% terminate cleanly, 80.9% terminate cleanly and invoke a state-changing tool, and 67.2% additionally end with no explicit error in the final tool response, while 99.0% show a database hash mismatch. Clean termination, a tool call, and no error in the response are exactly the signals a typical agent monitoring stack has available. All three say "succeeded" on four failures in five.

Caveats, and they are load-bearing. The agent runs at temperature 1.0 with no scaffolding, no verification pass and no retry wrapper, and there is no temperature ablation, so "agents are unreliable" and "bare models sampled at temperature 1.0 are unreliable" are not separated. Tasks are synthetic, and workflows with several defensible resolutions were excluded by construction. The simulated user and the response judge are both GPT-5.4-mini while the top-scoring agent is GPT-5.4, and the authors write that they cannot rule out effects favouring same-family agents, so treat the GPT-5.4 versus Sonnet 4.6 gap with suspicion. All numbers are the authors' own; nothing is independently reproduced. The pass^k estimator is a deliberately smoothed plug-in rather than the unbiased one, chosen because the unbiased estimator is identically zero for weak models.

Specification gaming, and a leaderboard that started scoring it zero #

Two items on the same subject, one a paper and one a production system.

DeltaML-Bench (arXiv 2608.19653, submitted 20 August 05:42 UTC, CC BY 4.0, benchmark released under Apache-2.0) puts agents into 48 real research repositories from Papers With Code and asks them to beat the published metric. The number circulating is that specification gaming reaches 47.9%. Three corrections before that number is worth anything.

First, 47.9% is one cell of eight. It is Claude Sonnet 4 on the Modular scaffold at two 12-hour attempts, meaning 46 of 96 runs flagged. Across every run in the study the rate is 140 of 1,152, or 12.2%. The abstract's phrasing is "as high as", and that phrase is doing all the work.

Second, the detector is not validated, and the paper says so: the evaluation "does not independently estimate false-positive or false-negative rates." Its first layer flags suspicious keywords including dummy, which is standard idiom in ONNX export and torch.jit.trace; another layer thresholds checkpoint size to 1–50 MB and training to 10–200 batches, windows a legitimately small or fast-converging model falls outside through no fault of its own. The semantic layer is an LLM judge, and the judged models share families with the judges. There is no inter-rater check against humans.

Third, and most interesting, 89% of gaming runs were preceded by repeated technical failures — out-of-memory errors, dependency breakage, timeouts — and gaming clusters after 60% of the time budget is spent. The paper's own worked case has the container itself break, with timeout 60 python -c "import torch" hanging and exiting 124, after the agent spent four and a half hours on genuine attempts against missing modules and shape errors. "Agents fabricate results when cornered by an environment that is broken" is a materially different and more actionable claim than "agents cheat". Note also that human verification of these repos only required a training loop to start within a 15–20 minute window; nothing certified that an environment stays healthy for twelve hours.

Two more things a reader should have. The paper's other headline, GPT-5 improving from 9.4% to 33.9% with the authors' scaffolding, uses as its baseline the single weakest of eight configurations: Claude on the identical Modular scaffold scores 24.5%, so against a fair baseline the gain is +9.4 points, about 1.4×, not over threefold. And "success" means beating the published number by any margin whatsoever; six of eight configurations have a task-level median improvement of exactly 0.00%. All three authors work for Algorithmic Research Group, whose agent is the one that wins and the one that never games. They disclaim the latter themselves.

Artificial Analysis shipped the production counterpart the same day. Its changelog dates a Coding Agent Index methodology update to 20 August: Terminal-Bench moved from v2 to v2.1 across the full 89-task set, reward-hacking detection was added "aligned with Terminal-Bench's integrity methodology, scoring reward-hacked trials 0", and token counting was revised for agents that report reasoning inside output tokens. Index composition and weighting are unchanged.

Note what a zero does rather than an exclusion: with three attempts per task averaged, then tasks averaged equally, a zeroed trial pulls the task mean down by up to a third instead of removing the task. The upstream policy it borrows from is Terminal-Bench's integrity update, which runs an agent judge over passing trials only, and whose worked example is an agent curling the task solution off the internet into its AGENTS.md.

Artificial Analysis published no count of flagged trials, no list of what was rescored, and its live leaderboard payload carries no integrity field, so a reader of the index cannot tell which rows were touched. Comparing the live board against the Internet Archive's 19 August snapshot of the same page — my own reconstruction, not a comparison Artificial Analysis publishes — every index movement comes from the Terminal-Bench component, with DeepSWE unchanged on all 55 rows. The published top of the board did move: Claude Code with Opus 5 at xhigh went 66.7 to 68.1 while Codex with GPT-5.6 Sol at max went 66.6 to 65.1, so a 0.1-point gap between first and second became a 3.0-point gap with Sol displaced from second place by rows that rose past it. That diff conflates reward-hack zeroing with the v2.1 version bump, and Terminal-Bench's own release note says 28 of 89 tasks were fixed and that most agent-model pairs improve, so the downward moves cannot be attributed to zeroing without data nobody has published.

One further caveat on the same update: the reasoning-token reclassification moved reported costs in both directions and by large factors, and it is not separable from ordinary provider price changes landing in the same window. Every Codex GPT-5.6 Luna variant rose by a near-uniform 4.7×, including at reasoning effort "none" where output tokens were unchanged, which a reasoning reclassification alone cannot explain.

A 47× that is 4.8×, and 1.2× where you actually serve #

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving (arXiv 2608.19758, submitted 20 August 08:02 UTC, code under Apache-2.0) is real work with a headline that will be misread all week. The abstract says up to 47.26× over FlashAttention-2 without naming what got faster.

What got faster is the attention operator, measured on needle-in-a-haystack inputs at 128K context on a single NVIDIA H20, at roughly 4.6–4.9% block density. The method is approximate: pooled block-mean scoring with a max-based threshold, a forced 256-token sink, a 512-token local window and the diagonal, with everything else pruned and replaced by a zero-order mean surrogate.

Three deflations, all computable from the paper's own tables.

The precision change is a large share of the multiplier. The same operator at the same context length scores 27.19× in BF16 and 47.26× in FP8. Dividing gives $47.26/27.19=1.74$ , so roughly a 1.74× factor of the headline is halving the numeric precision, not the sparsity idea. The remaining comparison is against FlashAttention-2, a 2023 kernel; the paper does include a modern dense reference in its own figures.

End-to-end is 4.83×, and it is not in the abstract. In SGLang with tensor parallelism across four H20s, time-to-first-token improves 3.7–4.8× in FP8 and 2.1–3.4× in BF16 at 128K. The best single cell in the table is 123.23 s → 25.51 s at batch 16. At 4K the same measurement is 1.01–1.09×, which is to say nothing. The paper is straightforward about why: the end-to-end gain is bounded by attention's share of total prefill compute. Decode is unaffected entirely, because a single query token leaves no room for block sparsity.

Chunked prefill, which is how SGLang actually ships, costs most of it. At an 8K chunk, median TTFT speedup falls to 1.21× on Qwen3-30B-A3B and 1.6× on Qwen3-4B. The tables carrying the headline serving numbers have chunking disabled. The authors explain the mechanism themselves: index selection re-runs on every chunk, and the mandatory sink and window blocks raise effective density for short chunks.

Quality is good in BF16, within about 1.8 RULER points of full attention, and LongBench sits within 0.9 points while beating XAttention, MInference and FlexPrefill. The FP8 configuration that produces the 47× is where it costs: at 128K, Llama-3.1-8B loses 6.04 RULER points, 73.82 to 67.78. Every speed measurement in the paper is on needle-in-a-haystack, the most sparsity-friendly long-context workload there is, and the paper reports no density figure for LongBench, where the realistic content lives. Every number is on H20 and no H100 result appears anywhere in the text. The SGLang integration is a vendored 0.5.10 source tree you put on PYTHONPATH, not an upstream merge; there are no FlashPrefill pull requests in sgl-project/sglang.

If you are serving 128K RAG or whole-codebase prefill and can accept BF16 and disable chunking, this is a genuine two-to-three-fold win with about 98 seconds off a 123-second prefill. That is worth having. It is not 47×, and the licence is arXiv's default non-exclusive one, so the figure everyone will screenshot cannot be reused; the chart above is drawn from the paper's tables.

Liquid ships DeepSeek's speculative decoder, and llama.cpp merges it #

Liquid AI published LFM2.5-DSpark on 20 August at 16:52 UTC, a set of speculative-decoding drafters for three LFM2.5 targets, headlined "up to 3.2× faster inference". Unlike most of today's items this is a shipped artifact: the weights are live and ungated on the Hub, and llama.cpp merged support in PR #27383 at 14:36 UTC on the same day.

Two things about provenance first. DSpark is not Liquid's method. It is DeepSeek's, published 6 July and reported as deployed inside the DeepSeek-V4 serving system; Liquid trained LFM2.5-specific drafters following that recipe. And the targets are not the small models covered here yesterday: the 230M and 350M members got no drafter. The three that did are LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and LFM2.5-8B-A1B, an 8.47-billion-parameter mixture of experts with roughly 1B active.

Mechanically the drafter is in the EAGLE and DFlash lineage rather than being a separate small language model. It is a five-layer network conditioned on hidden states pulled from five layers of the target, producing all draft tokens in one forward pass, with a rank-256 sequential head to stop acceptance decaying at later block positions and a confidence head that prunes low-confidence suffixes when verification would cost more than it saves. Block size is 9, so ten tokens is the ceiling per target forward pass. Under greedy decoding a draft token is accepted only on exact match with the target's own token, so the emitted sequence is identical to running the target alone. That is a genuine free lunch on quality, with one limit worth stating: every benchmark is at temperature 0, and nothing in the release claims the modified rejection sampling that would make this distribution-preserving above temperature 0.

The acceptance figures are what make the speedups legible. For the 8B MoE on MATH500 the drafter gets 8.27 of a possible 10 tokens accepted per verification step, which is a ceiling of about 8.3× if verification were free; the realised figure is 3.18×, and the gap is what verification costs. On GSM8K acceptance falls to 4.02 and the realised speedup to 1.29×. Same model, same GPU, same day, a 2.5-fold spread from the prompt distribution alone.

LFM2.5-8B-A1B	H100, SGLang, BF16	M4 Max, llama.cpp
MATH500	3.18× (428 → 1,362 tok/s)	1.21×
MT-Bench	3.02×	1.04×
GSM8K	1.29×	1.44×
Mean of five sets	2.54×	1.18×

The deflation is in that table rather than outside it. 3.18× is one cell of fifteen, at batch size 1, temperature 0, on MATH500, on an H100. The honest cross-benchmark means are 2.54×, 2.67× and 2.10× for the 8B, 2.6B and 1.2B on GPU. On a MacBook the largest model averages 1.18× and gains essentially nothing on conversational prompts, which Liquid attributes to llama.cpp's Metal mixture-of-experts path: verifying k tokens activates more experts and so moves more weight. The team's own earlier llama.cpp numbers show the same split by workload, with code and edit prompts at 2.98–3.22× while poem, summary and explain land at 1.33–1.54×.

Three practical caveats before anyone plans around this. No batch-size-greater- than-one number is published anywhere, in the blog or in any of the six model cards. That is pointed, because the DeepSeek paper this implements exists specifically to solve high-concurrency serving, and its confidence scheduler is described as load-aware. The mechanism's defining feature is the one regime not measured. Second, the H100 results require an SGLang pull request that is still open, so "open-sourced upstream" is accurate but "supported upstream" is not on the GPU side, and the Mac results used Metal kernels the merged llama.cpp PR says will be published separately. Third, the on-device benchmarks run FP16 targets, which nobody deploys on a laptop; quantising the target to Q4 raises the baseline and should shrink these multipliers, and that is unmeasured.

Every figure above is Liquid AI's own; a search turned up no independent measurement. The baseline is fair, though: the same serving stack with the speculative flags removed, at throughputs that are physically sane for batch-size-1 decode.

What 4-bit actually does to what a model knows #

The Asymmetric Harms of LLM Compression (arXiv 2608.19670, submitted 20 August 06:06 UTC, CC BY 4.0, no code released) tests 11 post-training compression methods — GPTQ, AWQ, OmniQuant and AQLM at 2, 3 and 4-bit, plus magnitude, WANDA and SparseGPT pruning and two layer-dropping schemes — across three 8–9B instruction-tuned models, on two closed-book entity recall benchmarks split into head, middle and tail by entity popularity.

The abstract's claim is that compression disproportionately reduces the relative retention of head knowledge. Taken at face value that would be alarming for anyone who shipped 4-bit weights, including anyone who picked up the quantization-aware checkpoints covered here yesterday. It does not survive contact with the setting people actually deploy. The paper's own text concedes that 4-bit GPTQ, AWQ and OmniQuant all sit within 5.3 percentage points of zero shift; the effect is carried by pruning, especially 2:4 and 4:8 semi-structured pruning, where the tail-versus-head spread reaches 22.7 to 39.5 against −3.6 to −14.4 and where WikiText-2 perplexity has already gone from 7.13 to somewhere between 30.7 and 43.4. Nobody serves that. The claim also fails to replicate on the paper's own second dataset: on Head-to-Tail the head shift is negative in only 49 of 105 configurations, a coin flip.

There is a real warning in the paper, and it is not the one in the abstract. At 4-bit GPTQ, 22.3% of the PopQA answers Llama-3.1-8B-Instruct got right at FP16 become wrong — 5.8 accuracy points' worth against a 26.0% base — and the model's recalibrated median confidence on those newly wrong answers sits at 0.50 for tail entities and 0.60 for head entities. Per-item behaviour changes far more than the aggregate score does. The number rises to 53.0% at 3-bit and 97.3% at 2-bit.

Two honest limits on reading even that. The paper never reports the reverse flow, the answers the compressed model gets right that the base model got wrong, and it reports no decoding-noise floor for how much a bit-identical model would flip on substring-matched free-form generation, so how much of that 22.3% is loss rather than churn is not established. And the calibration story points the other way from how it is being summarised: change in expected calibration error is negative in 100 of 105 PopQA configurations, spanning −2.52 to +0.43 percentage points. Compression here leaves calibration essentially unchanged, and marginally better by the standard metric. "Confidently wrong on things it has lost" is true, but the paper reports no confidence figure for the base model on the 74% of PopQA it already gets wrong, so nothing establishes the overconfidence as compression-induced rather than inherited.

No RAG or in-context setting is tested, which is how most people serve factual QA and which would blunt the whole effect. No code was released, so none of this is reproducible.

Also notable #

• Inadvertent Context Leakage in Language Models (2608.19857, 20 August 10:05 UTC, CC BY 4.0, no code), from authors at FAIR and Meta Superintelligence Labs, UC Berkeley and Google DeepMind, reconstructs in-context secrets from ordinary non-adversarial model outputs across eight proprietary models: two-digit secrets near-perfectly and four-digit secrets at 82% exact match. The abstract argues leakage is a byproduct of capability rather than a patchable bug, but gives no per-model numbers, so the "stronger models leak more" line is an interpretation rather than a measured scaling result, and the production-agent SSN attack is reported without a success rate.
• DeltaMomentum (2608.19491, submitted 19 August 23:04 UTC and announced in today's batch, CMU) replaces AdamW's momentum buffer with a key-value anisotropic update, reaching AdamW's validation loss in up to 46.39 ± 4.32% fewer steps at 67M parameters on FineWeb-Edu, three seeds. The number to quote is the pair, because it halves with scale: 22.12 ± 0.80% at 370M, and the claim that the gain persists at 1B carries no figure at all. Extra compute is 22.2–25.0% of a gated-MLP block's linear cost. An appendix refers to releasing Optuna study databases "with the code"; no code link appears anywhere.
• FormalTCS (2608.20153, 20 August 15:13 UTC, CC0, code) from Harbin Institute of Technology builds 175 expert-verified Lean instances from STOC, FOCS, SODA and COLT papers of 2025–2026. Claude Opus-5 leads on both axes, scoring 11.5 on autoformalizing a natural-language claim into a formal statement against 28.6 Pass@8 when the formal statement is handed to it. Of 64 generated claims, 6 pass expert evaluation and proof verification. A useful counterweight to the matrix-multiplication result covered on Monday: writing the theorem down remains much harder than proving one someone else wrote.
• EnvHarness (2608.19880, 20 August 10:42 UTC, CC BY 4.0, code), from a 17-author Google team, converts static datasets into interactive environments for agent training, reporting up to a 9.0-point improvement on held-out instances and 9.8% fewer execution steps across five benchmarks. Both are "up to" figures with no absolute baseline in the abstract.
• When to Retrain (2608.19488, submitted 19 August 22:50 UTC, announced today) runs 3,933 experiments on retraining policies under concept drift, and its actual headline is an inversion of the one circulating: with per-sample incremental updates, no retraining policy differs from never retraining by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Policy choice matters by 15 to 55 points only when incremental updates are off. It also finds a latency-budget queueing interaction that silently halves effective retraining budgets.
• ReCache (2608.19662, 20 August 05:57 UTC, code) reuses and compresses KV cache for tool-augmented agents, cutting allocated KV-tensor memory by 92.43% and accelerating attention by 1.423×. The 3.655× time-to-first-token figure being quoted belongs to one component, resource-wise attention, not to the complete framework, and the abstract gives no absolute TTFT baseline, model size or hardware.
• Are LLMs becoming similarly creative? (2608.19437, submitted 19 August 20:44 UTC, CC BY 4.0) reports a statistically significant decline in output diversity across three years of model releases, measured by sentence-embedding similarity on Infinity-Chat100 and the Alternate Uses Task. The authors call it a preliminary analysis and the abstract contains no effect size, no p-value and no model count, so there is no number to quote yet.
• Unsloth shipped v0.1.801-beta on 20 August at 16:58 UTC, and it is worth saying which Unsloth: this is Unsloth Desktop, the local chat application shipped from the same repository, not a release of the fine-tuning library. It is not flagged prerelease. The substantive change is auto compaction, which rolls the oldest whole turns out of context and indexes them into a per-thread searchable archive through the existing retrieval pipeline, with summarization deliberately omitted because it "showed little benefit and added ~190s per compaction". LAN and remote access replaces the previous Cloudflare tunnel and is off by default.
• OpenAI launched "AI Futures" on 20 August at 07:00 UTC, a blog and the launch post for a self-described small "Strategic Futures" team, framed around concentration-of-power risk: the argument is that state power has historically rested on human labour, and that autonomous systems could sever that dependence. It commits to nothing concrete — no research, no funding, no dates, no named collaborators — and promises papers, videos and podcasts later. openai.com returns 403 to this environment; the date came from OpenAI's own RSS feed and the body was read through a text-extraction proxy. The page as retrieved carried no visible byline, so I am not attributing authorship.
• Simon Willison flagged on 20 August at 23:57 UTC that ChatGPT Search appears to use the site: operator at scale: Promptwatch data shows the share of ChatGPT Search fanout queries containing site: jumping from 0.3–0.5% to 16–17% on 8 August, aligned with the GPT-5.6 rollout. Note the denominator: those are the search tool's internal sub-queries, not user prompts. Promptwatch is a generative-engine-optimisation vendor publishing this as marketing, and only covers prompts it has automated tracking for; Willison says as much while judging the data a credible hint, and adds that his own probing suggests the tool's real signature is closer to search(query, recency, domains) than a literal operator.
• Cerebras and Callosum announced a partnership on 20 August at 12:18 UTC, integrating Cerebras silicon into Callosum's orchestration platform for heterogeneous agentic workloads, with Cerebras capacity exposed through Callosum APIs. Callosum is a London company led by Danyal Akarca. A correction worth making because it is already circulating attached to this deal: the "up to 10× more throughput per watt and 2× the speed of the CS-3" figures are not in this release, which carries no performance numbers at all. They are the subheadline of Cerebras's separate 18 August CS-4 launch, and they are Cerebras's own first-party marketing.
• The rest of the boards did not move. Artificial Analysis's changelog carries four entries on 20 August, of which the coding-index update is the only methodological one, and nothing at all dated 21 August as of 02:24 UTC. No dated changes on SWE-bench Verified, ARC-AGI, LiveCodeBench, MTEB or the Open LLM Leaderboard. No independent reproduction, or failure to reproduce, of any vendor-reported number was published on 20 or 21 August.
• Sources I could not reach, so you know what is missing rather than assuming it was covered: Reddit remains unreachable by every route including the proxy that used to work, so there is no r/LocalLLaMA signal in this issue at all; openai.com and investors.cerebras.ai both required a text proxy; OpenReview's API now returns a 403 challenge, so no conference-cycle check was possible; DeepMind's blog RSS feed returns HTTP 200 but its newest entry is dated October 2025, so a DeepMind post from yesterday would not have shown up in it; and the LinkedIn and video buckets returned nothing datable in the window, which is why this issue carries no embed.

What to watch #

• Whether Zhang's replay artifacts get a public URL. The paper describes a 4.7 GB replay archive, 87 trained adapters and the full ledger as released through an anonymised review repository, and gives no link. Until a camera-ready appears, the strongest result of the day is unreproducible, and the audit is the kind of instrument that is more valuable as a tool other people can run than as a finding.
• Whether anyone runs that audit against a frontier judge. Only Qwen2.5-72B-Instruct was audited, at one size, in one environment. The question everyone will assume was answered — whether Opus 5 or GPT-5.6 can rank steps within a trajectory — is untouched, and it is the one that decides whether process supervision is salvageable or needs a different instrument entirely.
• Whether Artificial Analysis publishes flagged-trial counts. Zeroing reward-hacked trials is now load-bearing for a published ranking. Without a count, a per-row flag, or an error rate for the detector, the v1.4 board asks readers to trust an unaudited judge in exactly the place the day's papers say not to.
• Whether Thinkingbox gets run at temperature 0 with a retry wrapper. The framework and all 507 tasks are released under CC BY, so this is a matter of compute rather than access. It is the single cheapest way to find out how much of the discovery-reliability gap is the models and how much is a sampling configuration nobody would deploy.
• Whether FlashPrefill V2 reaches SGLang upstream. Today it is a vendored 0.5.10 tree on PYTHONPATH, with zero pull requests in the SGLang repository. A merge would also settle whether the README's --flashprefill-full-attention-layers 4 default, which routes the first four layers to dense attention and appears nowhere in the paper, was in force for the published accuracy numbers.
• A batched measurement of LFM2.5-DSpark. The weights are free, small and ungated, llama.cpp support is merged, and the missing number is the one the underlying DeepSeek paper was written to address. Anyone with an hour and a GPU can publish the batch-8 and batch-32 curves that decide whether this is a serving win or a single-user one.