AI Brief, 8 September 2026: two evaluations swapped, and the top two nearly tied

Artificial Analysis published Intelligence Index v4.3 on 7 September, three days after it published v4.2. Two of the eleven evaluations in the composite changed: Terminal-Bench moved from v2.1 to v4.0, and the τ³-Banking agentic benchmark was replaced by AutomationBench-AA, the firm's own implementation of Zapier's business-workflow benchmark run against a held-out set of 657 tasks. Every leading score fell for the second version running, this time by between 1.85 and 6.17 points against six to ten points last time, and the order of the top seven models did not change at all.

What changed is the distance between the top two. On the v4.2 scale published on 4 September, Claude Fable 5.1 led GPT-6 Astra by 2.10 index points. On v4.3 it leads by 0.56, and Artificial Analysis's own summary of its results now reports both models at 53. The two swapped evaluations are enough on their own to account for that: Astra beats Fable 5.1 on both of the new ones, and gains 1.75 index points relative to it purely from the substitution, against an observed narrowing of 1.54. Everything else in the index moved slightly the other way.

That is three scales in four days, and still nothing said about whether numbers from any of them can be set beside each other. The word "comparable" does not appear anywhere in the v4.3 announcement. The closest it comes is a line saying each change "stands on its own merits" — a statement about why the changes were made, not about what the resulting numbers mean next to last week's. This account asked for that statement on the 5th, the 6th and the 7th, and asks again.

Elsewhere in the window, the team that runs kernel.org has published hard measurements of what AI scraper traffic costs it: of 90 CPU cores across five sites, 14 to 16 do nothing but render git commits as HTML for bots, and by its own generous accounting about 2% of requests to git.kernel.org are plausibly human. That post is dated 29 August and this account is late to it. And three separate Anthropic stories broke on Monday, none of which the company has commented on.

• Intelligence Index v4.3 swaps Terminal-Bench v2.1 for v4.0 and τ³-Banking for AutomationBench-AA; private held-out sets rise from 40% to 45% of index weight, category weights unchanged.
• Every leading model fell again: Claude Fable 5.1 by 3.39 points, GPT-6 Astra by 1.85, Grok 4.6 by 6.17. No model changed rank.
• First and second are now 0.56 index points apart, about eleven flipped trials out of Terminal-Bench's 198.
• kernel.org reports roughly 6 million daily requests for random git commits, a third of which now solve its proof-of-work challenge, and about 2% of traffic that looks human.
• Anthropic reportedly walked away from a roughly six-billion-dollar acquisition of Decart; The Information puts its signed compute commitments at 517 billion dollars; and Matt Clifford resigned as ARIA chair after joining the company. There is no first-party statement on any of the three.
• Artificial Analysis added MiniCPM5-2B at an estimated index of 14.28. Tencent's Hy4 preview still has no value at all, a sixth issue running.

Two evaluations swapped, and the leaderboard's top gap nearly closed #

The Intelligence Index is a weighted average,

$$\text{Index}=\sum _i{w}_i{s}_i$$

where $s_i$ is the model's score on evaluation $i$ as a percentage and $w_i$ is that evaluation's published weight. Version 4.3 changes exactly two of the eleven terms: Terminal-Bench v2.1 becomes v4.0 at the same 10% weight, τ³-Banking becomes AutomationBench-AA at the same 5%, and category weights are untouched at Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%. Because the new AutomationBench set is held out, the share of index weight sitting on private questions or answers rises from 40% to 45%.

Both replacements are harder. Terminal-Bench v4.0 is 66 tasks, run three times each with the average pass@1 reported, and Artificial Analysis says it recalibrates compute and time allowances and rewrites task instructions, environments and verifiers. AutomationBench-AA is 657 workflows across finance, HR, marketing, operations, sales and support, at version 1.0.6 of Zapier's benchmark. The scoring rule is the interesting part and it is the firm's own: partial credit is awarded for completed objectives, but any guardrail violation drops that task to zero. That is why it carries the -AA suffix, and it is a large choice. On the partial-credit score Astra reaches 68.5%; on the stricter count of workflows completed cleanly with no violation it reaches 41.6%. The index uses the first number.

The effect on the leaderboard is a second uniform contraction with no reordering.

The size of each drop tracks how a model does on the two new evaluations. Astra lost least, 1.85 points, and it leads both replacements: 59.1% on Terminal-Bench v4.0 against 52.0% for Fable 5.1 and 49.0% for Claude Opus 5, and 68.5% on AutomationBench-AA. Grok 4.6 lost most, 6.17 points, and manages 21.2% on the new Terminal-Bench.

That is enough to make the top of the board arithmetic rather than judgement. Terminal-Bench v4.0 is 66 tasks run three times, so every published figure is a count out of 198 — Astra's 59.09% is exactly 117 trials, Fable 5.1's 52.02% is 103, Opus 5's 48.99% is 97.

trials = 66 * 3                        # 198 attempts, average pass@1
astra, fable = 117 / trials, 103 / trials
one_trial = (100 / trials) * 0.10      # 0.0505 index pts at Terminal-Bench's 10% weight
(53.3738 - 52.8141) / one_trial        # 11.08 — the gap between 1st and 2nd
0.10 * (astra - fable) * 100           # 0.707 — Astra's Terminal-Bench edge, in index points

Astra wins Terminal-Bench by fourteen trials, worth 0.707 index points, and wins AutomationBench-AA by 9.1 points, worth another 0.456. Together that is a 1.16-point relative gain on the new evaluations, against a 0.588-point deficit on the two they replaced — a swing of 1.75 index points in Astra's favour, when the observed narrowing of the gap was 1.54. The substitution more than accounts for the whole of it.

What AI crawlers cost kernel.org #

This one is late: Konstantin Ryabitsev, who runs the Linux kernel's public infrastructure, published Creepy crawlies on 29 August, and it resurfaced when Simon Willison linked it late on 7 September. It is the most concrete published accounting of scraper load on a major open-source host, and it did not get covered here when it appeared.

The measurement: of 90 cores across five geographically distributed nodes, 14 to 16 are constantly occupied rendering git commits as HTML for scrapers — about 20% of total capacity, and spiky rather than flat, because the swarms arrive in waves. git.kernel.org takes roughly 6 million requests a day for arbitrary commits. Two thirds are turned away at the door; the remaining third get through. On Ryabitsev's own generous assumptions, about 2% of what reaches the service is plausibly a human.

The mechanism is the part worth understanding, because it is self-inflicted on the crawler's side. Nothing about this workload requires rendering: a git clone of linux.git transfers the entire history once, in a packed format the server hands over cheaply. Instead the scrapers walk the web interface, requesting commits, patches, plain renders and diffs between arbitrary pairs of commits, each generated on demand. linux.git holds about 1.48 million commits and git.kernel.org carries some 922 forks of it, so the diff-between-any-two-commits endpoints alone expose an effectively unbounded URL space, and a crawler enumerating URLs cannot tell it is re-downloading the same repository several hundred times.

The defensive history is an escalation in which each rung lasted less than the one before. Blocking by user agent worked until the bots sent browser strings. Blocking by network worked until the traffic moved onto residential and mobile addresses rented through proxy SDKs embedded in consumer applications. The current defence is Anubis, a SHA-256 partial-collision challenge the client must solve before being served; at difficulty 4 the bots simply stopped, then learned to solve it, and difficulty 5 now takes a few seconds on a phone and is being solved by a third of arriving requests. The next step is not a better filter but a smaller surface: turning off features and gating expensive operations for anonymous callers, while continuing to offer bulk downloads to anyone who asks.

The post names nobody. There is no company, product, user agent or network identified anywhere in it, and no claim about whose crawlers these are. Any secondhand version attributing this load to a specific lab is going beyond what was published.

Three Anthropic stories, none of them on the record #

Monday produced three separate reports about Anthropic and no statement from Anthropic about any of them. They are worth taking together because the evidence behind each is different, and that difference matters more than the headlines.

Decart. Bloomberg reported on 8 September that Anthropic has walked away from an acquisition of the Israeli video-model startup Decart, which the same outlet had reported on 13 August as talks valuing it at about six billion dollars. The report rests on people familiar with the matter, states that nothing had been finalised — so there was no signed deal to break — and records that both companies declined to comment. Anthropic's newsroom carries nothing on it. The original is paywalled; this account follows a licensed syndication of the wire text carrying the same dateline.

517 billion dollars of compute. The Information published a piece on 6 September putting Anthropic's compute commitments signed over eleven months at 517 billion dollars. Every other version, including The Decoder's on 7 September, relays that single paywalled piece rather than reporting it independently. Anthropic has never disclosed a total and has not disclosed one now. It is private and files nothing itself, but a search of SEC filings across the window turns up no 8-K from any counterparty corroborating a commitment on that scale either. The figure is secondhand, single-sourced and unconfirmed at any primary document.

ARIA. The best-supported of the three. Matt Clifford resigned on 7 September as chair of the Advanced Research and Invention Agency, the UK's high-risk research funder, which he had chaired since 2022, after taking a role at Anthropic as managing director for international affairs announced on 2 September. He stays until 6 November while a successor is found. Dame Chi Onwurah, who chairs the Commons Science, Innovation and Technology Committee, had called the overlap a clear conflict of interest. Several outlets carry the same facts and timings — the Guardian's account is stamped 14:42 UTC on the 7th — but neither ARIA's own site nor gov.uk had published anything, so even this rests on Clifford's words as relayed by the press.

Also notable #

• Google Research is flying contrail avoidance with Cathay Pacific. The trial, posted 08:00 UTC on 7 September, pushes predicted contrail-forming altitude bands to the flight deck over in-flight Wi-Fi so crews can make small planned altitude changes. More than 100 flights were targeted and more than 80 flew an avoidance route; Google's own satellite analysis puts the reduction in warming impact on those flights at roughly 40%, with the Hong Kong–Singapore corridor contributing over half the total. No model is named, no paper accompanies it, and every figure is Google's own.
• A 2.6-billion-parameter model got an independent score. Artificial Analysis added MiniCPM5-2B, Apache-2.0 with a 128k context, whose weights went up at 11:35 UTC on 6 September. Its index value of 14.28 is flagged as estimated rather than fully measured — a distinction the firm publishes but does not define — against a stated median of 6 for open-weights models of similar size.
• SemiAnalysis published its own TPU-versus-Blackwell numbers at 20:00 UTC on 7 September, reporting Ironwood getting 50.4% more tokens per dollar than a B200 at 20 tokens per second per user on external cost assumptions. These are SemiAnalysis's measurements on its own benchmarking platform, not Google's; Google has published nothing under that name, and the article's paywall falls immediately before the cost breakdown the percentages are derived from.
• OpenAI announced a Ukrainian journalism programme on 7 September with WAN-IFRA and the Association of Independent Regional Press Publishers of Ukraine: masterclasses, a cohort programme for ten newsrooms, and API credits. No monetary figure is attached to any of it.

What to watch #

• Whether Artificial Analysis ever states a comparability position, now asked here for a fourth consecutive issue. The question has got harder rather than easier: there are three scales in play instead of two, and both recent releases are billed as instalments of a v5 still to come.
• Whether anyone reproduces AutomationBench-AA. The rule that a single guardrail violation zeroes a task is the firm's own addition to Zapier's benchmark, and the gap between the two metrics it produces is large — 68.5% partial credit against 41.6% clean completions for the same model. Which of those is the right number to weight at 5% of an intelligence index is a live question.
• Whether the top two separate again. Fifty-six hundredths of an index point is about eleven Terminal-Bench trials out of 198, and the next model added or evaluation swapped could move it either way.
• Whether kernel.org's feature removals land, and whether any comparable host publishes numbers. The measured-cost argument is currently one sysadmin's post; a second set of figures from another large public host would turn it into evidence about the ecosystem.
• Tencent's Hy4 preview still has no independent index value, a sixth issue running. Every published figure for it remains the vendor's own.