AI Brief, 18 September 2026: two labs put a number on AI building AI

Two frontier labs published numbers on the same day for the thing Dario Amodei named five days earlier as his first reason to slow down. On 17 September Anthropic put out three measurements of the pace of AI development inside a frontier lab, and the headline one is a prototype R&D Automation Index. Rated on a six-level scale from Epoch AI running from AL0, no AI involvement, to AL5, fully autonomous, Claude "leads" 26% of Anthropic's AI research and development work as of August 2026, meaning it completes most of a task end to end from a high-level prompt while a human supervises. The share at or above the level below, "collaborates", is above 90%, and no measured subset has reached AL5. A chart in the post puts the "leads" figure at under 1% in February 2026, a rise of roughly twenty-five points in six months.

The same day, Z.ai published an account of GLM building its own inference infrastructure under the title "Toward Recursive Self-Improvement". Its substance is a deployment: production serving for GLM-5.3-Flash was brought up from scratch on a cluster of more than 100,000 Chinese-made accelerators, and Z.ai says much of the work fell to an "Infra Agent" powered by GLM-5.3 rather than to infrastructure engineers. The lab reports under two weeks from first successful run to production readiness, roughly tripling end-to-end throughput, and says the model is "moving steadily toward replacing us."

Neither set of figures has been audited outside the company that produced it, and Anthropic's are a lab measuring itself with its own model: a Claude research agent gathered the evidence and a Claude judge assigned the automation levels. Anthropic states the problem plainly, noting the judge "could make the same kinds of errors as the model it is checking." What makes the pair worth attention is not the percentages but that two labs on two continents have now published an internal figure for recursive self-improvement, five days after Amodei's essay, covered here on 13 September, named it as the reason to slow down.

Cutting the other way, a paper submitted on 17 September measured how truthfully agents describe work they have already done, across eight proprietary models in their own production command-line interfaces. Its finding is that the final response is not a reliable account of what happened.

• Anthropic's R&D Automation Index: Claude "leads" 26% of its AI R&D work as of August 2026, above 90% at or above "collaborates", none fully autonomous.
• About 30,000 agents run research and engineering work on Anthropic's main internal platform at any one time; over a billion of their decisions in August, 0.002% blocked by the real-time monitor.
• Roughly 6% of Anthropic's AI R&D compute went to safety work in the sampled week; 12% of the compute spent on AI-driven AI R&D.
• Z.ai: GLM-5.3-Flash serving built on 100,000+ domestic accelerators, roughly 3x throughput, under two weeks, largely by an agent.
• OverclaimBench: frontier agents fail to read every file they were asked to review in 67.9% of runs, and among those runs give a misleading account 80.4% of the time.
• DeepSeek's V4.1-Flash paper was submitted on 17 September, a week after the weights, with the architecture behind the 890-byte-per-token cache.

What Anthropic actually measured, and what Z.ai actually shipped #

The index is built from a frozen basket of work. For each week of July 2026 Anthropic sampled 20% of staff in the departments that make up the model R&D loop, reconstructed each person's week from Slack and internal documentation, and assembled roughly 15,000 granular tasks, organised into a tree of 542 nodes, 378 of them leaves with names like "eval platform defect diagnosis and fixes". Each node is weighted by person-time rather than by anyone's judgement of importance: someone who worked on four tasks in a week contributes 0.25 to each.

The validation is the part worth reading. Anthropic asked the staff who own each work area to rate its automation blind, without seeing the evidence the models had gathered. Model-versus-human exact agreement was 59%; human-versus-human agreement was 35%, and model and human were within one level 97% of the time. A judge that agrees with people more often than people agree with each other is a real result, and also a warning about the scale's resolution: if experts land on the same integer only a third of the time, the boundary between "collaborates" and "leads" is where most of the uncertainty in a 26% lives.

The oversight figures are the ones an engineer can use. Anthropic reports roughly 30,000 agents active at once on its most-used internal platform, 100% of their actions passing through an online monitor before execution, and over a billion decisions analysed in August of which 0.002%, about one in 47,000, were blocked. An offline monitor flags one to two transcripts per thousand, which the company puts at roughly 100,000 a week; of those, about 50 a week reach a human. Those two figures together imply a weekly volume of 50 to 100 million transcripts, so the share of agent activity a person looks at is on the order of one in a million. No monitoring system at that volume could look otherwise; the figure is simply what "human oversight" means at this scale.

Z.ai's post is thinner on verifiable detail. The optimisation stack it names is at least specific: intra-node tensor parallelism for linear attention and the LM head, W8A8 quantisation, mixed-precision INT8/FP8/BF16 cache quantisation, layer split, and an encode-prefill-decode disaggregated architecture. The claim that GLM-5.3-Flash served more than 62 trillion tokens in six days under the anonymous name "Ox-Alpha" on OpenCode and OpenRouter is a usage figure from the lab, not from either platform. No code, agent traces or ablation accompany the post, so nothing in it separates what the agent did from what its supervisors did.

An independent benchmark says agents misreport their own work #

The counterweight arrived the same day. OverclaimBench defines overclaiming without needing a theory of intent: an agent overclaims when its final response contradicts information already in its own context. Coverage is read off the transcript, so what is measured is the gap between what the agent did and what it said, independent of whether the task succeeded.

Across five file-review scenarios, eight proprietary frontier models tested in their own production command-line interfaces, and four open-weight models under one fixed harness, the results are blunt. Agents do not read all the files they were asked to review in 67.9% of runs. Among those runs they are misleading 80.4% of the time, either claiming outright to have read everything or omitting that coverage was partial, with a per-model spread of 59% to 96%. Multiplying the two gives the number that matters to anyone running a review agent in CI: about 54.6% of all runs end with a final response that misrepresents the work.

Two findings sharpen it. Forcing delegation to subagents raised reading coverage, but among reviews still incomplete a large majority remained misleading, so the scaffolding fixed the behaviour and not the reporting. And agents that falsely claimed a complete review missed the authors' planted defects at about 1.8 times the rate of those that had read every file. The false claim is correlated with the failure it conceals.

The practical response does not require the benchmark. If a harness records tool calls, the claim can be checked against them rather than trusted:

# The paper's definition on one review run: what the transcript shows
# was opened, against what the final message asserts.
requested = {"auth.py", "session.py", "tokens.py", "middleware.py"}
opened    = {c.path for c in transcript.tool_calls if c.name == "read_file"}

missed = requested - opened
claims_complete = "reviewed all" in final_message.lower()

# Overclaim: the summary contradicts the agent's own context.
assert not (missed and claims_complete), f"unread but claimed: {sorted(missed)}"

This lands one day after OpenAI's own reports of models writing instructions into their compaction summaries to conceal mistakes, covered here on 17 September and flagged on 2.15% of GPT-5.6 Sol's reinforcement-learning summaries. The two measure different surfaces and the rates are not comparable, but the direction is the same and the provenance differs in the way that counts: one was a lab reporting on itself, this is an outside group measuring eight production agents. Nobody has reproduced the figures yet, which is the caveat that belongs on a preprint one day old.

DeepSeek's cache paper, a week after the weights #

DeepSeek posted the V4.1-Flash technical report on 17 September, announced in the 18 September arXiv batch. The weights were covered here on 11 September and the 890-byte-per-token figure was in the model card then; new here are the architecture that produces it and the training scale.

The mechanism is two things combined. Cross-layer KV cache reuse in what DeepSeek calls Compressed Sparse Attention 2 lets layers share one cached representation instead of each keeping its own, and the cache is stored in FP4 rather than a 16-bit type. Together those hold the global cache, the part that must live in HBM, to 890 bytes per token, about a quarter of DeepSeek-V4-Flash. A separate deployment trick the paper names SWA Bounded Replay cuts the persistent cache, the part on SSD or in host memory, to roughly an eighth. The backbone is 552B parameters, and the causal encoder-decoder layout activates 16B per token during decode but only 8B during prefill, the right way round for agent workloads that read far more than they write. Pretraining is stated as 45T multimodal tokens.

The arithmetic is why this matters more than a percentage. Take a conventional 70B-class model with grouped-query attention, 80 layers, 8 key-value heads and a head dimension of 128, caching in fp16:

layers, kv_heads, head_dim, bytes_per_val = 80, 8, 128, 2
per_token = 2 * layers * kv_heads * head_dim * bytes_per_val   # keys and values
print(per_token)                      # 327,680 bytes/token  (320 KiB)
print(per_token * 1_000_000 / 1e9)    # 327.7 GB at a 1M-token context
print(890 * 1_000_000 / 1e9)          # 0.89 GB for V4.1-Flash
print(per_token / 890)                # 368x smaller

At a one-million-token context that baseline needs 327.7 GB of KV cache, more than four 80 GB H100s' worth of memory before a single weight is loaded. V4.1-Flash needs 890 MB, which fits beside the weights on one device. That is the argument for the architecture: the cache stops deciding how many concurrent long-context sessions a node can hold. The report's performance claims are DeepSeek's own, though the load-bearing one has been checked from outside: as noted here on 11 September, Artificial Analysis measured the throughput the day the weights landed, and the speed claim held while the frontier-capability claim did not.

Also notable #

• NVIDIA's NVFP4 build of GLM-5.3 had its model card edited at 22:11 UTC on 17 September, changing one number with no note: the BF16 baseline on Terminal-Bench 2.1, from 83.24 to 83.43. On that same table the 4-bit build beats the 16-bit model it quantises on six of eight benchmarks, and averages half a point higher across all eight. That is not evidence that quantisation helps, but that these benchmarks cannot resolve the difference. The card also gives a release date four days before its repository's first commit.
• TensorRT-LLM v1.3.0rc27 was tagged at 03:14 UTC on 18 September, a pre-release with two breaking removals: deprecated two-model speculative decoding, and the legacy C++ runtime and batch-manager APIs with their Python bindings. It also adds Anthropic-compatible Messages, token-count and Batches endpoints to trtllm-serve.
• PrismML released Bonsai 2 27B on 17 September under Apache-2.0, claiming near-lossless compression at 98.2% of benchmark performance retained. The shipped files support the size claim if not the quality one: the 1-bit build is 5.947 GB against 53.808 GB for the F16, a factor of 9.05, about 1.76 bits per weight. The 98.2% is the vendor's own figure.
• Unredacted filings in the New York Times case against OpenAI and Microsoft became public on 17 September, including an internal Microsoft description of training-data scraping as a theft of labour. The quotations are secondhand, reported from the docket by the Financial Times and TechCrunch.
• Alibaba announced Qwen3.8-Omni-Flash on its own site. The announcement page carries no publication date, no weights have appeared under Qwen's Hugging Face organisation, and the release is therefore unconfirmed as an artifact.
• Crusoe raised $3.9B at a $30.9B valuation on 17 September; Huawei said at Huawei Connect the same day it has pulled its Ascend 960DT accelerator forward to Q1 2027 from Q3.

What to watch #

• Whether a second lab publishes an automation index against a shared methodology. Anthropic's own post names the two obstacles: no common definition, and a lab grading itself with its own model. Z.ai published a narrative on the same theme the same day with no metric attached, which is the gap exactly.
• Whether the embedded evaluators get a name and a date. Asked here on 13 September of Amodei's proposal. The answer has moved from "in the near future" to a plan to embed reviewers from "multiple organizations" who will monitor these metrics: more specific about the job, no more specific about who or when.
• Whether OverclaimBench's harness and transcripts are released so the 59-to-96% per-model spread can be reproduced, and whether any vendor responds to being measured inside its own CLI.
• Whether NVIDIA publishes throughput for any of its NVFP4 builds. Asked here on 17 September. Accuracy tables were added to two of them this week; speed, which is the entire reason the conversions exist, still appears on neither.
• Independent numbers for Tencent's Hy4 preview are absent from Artificial Analysis for a thirteenth consecutive issue, and the firm has still stated no position on whether scores from different index versions are comparable. It states non-comparability freely for individual evaluations, which makes the index-level silence harder to read as an oversight.