AI Brief, 24 October 2026: the comparison table that undersold its own model

Moonshot AI announced Kimi K3.1 Pro on 23 October, calling it "our most capable 3T-class open-weight model to date". The weights do not exist publicly. The post says when they will: "Full model weights will be released by Nov 4, 2026." Until then it is an API, live on Moonshot's platform and on OpenRouter at $2 per million input tokens and $10 per million output, with a 1,048,576-token context. A faster variant, K3.1 Pro Highspeed, ships alongside at a claimed two to three times the speed. The announcement reached 792 points on Hacker News.

The unusual part is what happened to the numbers. Moonshot's post puts K3.1 Pro at 65.7% on Terminal-Bench 4.0, ahead of GPT-6 Astra at 58.2% and "narrowly behind Claude Opus 5.5" at 66.4%. Artificial Analysis published its own measurement the same day and ranks it differently: it scores K3.1 Pro at 67.2%, with Opus 5.5 and GPT-6 Astra both at 59.6%. The independent number is higher than the lab's own, and it moves the model from third place to first. The reason is in Moonshot's footnotes, and it is a methodology problem that worked against Moonshot rather than for it.

Moonshot also did something no lab has done here before: it published the cyber-capability score of a model it decided not to release. On an in-house exploit-development benchmark, the checkpoint trained without any risk-reduction measures solves 90.5% of tasks; the shipped K3.1 Pro reaches 81.0%. That is a lab putting a number on what its safety intervention cost.

• Kimi K3.1 Pro, announced 23 October, is API-only until a promised 4 November weights drop. No repository for it exists under moonshotai.
• Artificial Analysis measures it at 55.1 on its Intelligence Index, second behind Claude Opus 5.5 (57.6) and ahead of GPT-6 Astra (52.4), and records 2,800B total parameters with 104B active — neither figure appears in Moonshot's post. It also classifies the model proprietary, with no licence and no weights URL, while recording its predecessor K3 as open-weight.
• Moonshot's own benchmark rows are self-run on one harness; comparator rows are "the maximum value reported in either their blog posts or the official leaderboard".
• On an internal exploit benchmark, removing cyber-related training data cut pass@3 from 90.5% to 81.0%.
• HobbyLM-1B Computer Use, a 14 MB adapter from a solo developer, shipped the same day with sha256-sealed test sets and reports failing both of its own pre-registered pass bars.

An open-weight model with no weights, and a score that beats the lab's own #

K3.1 Pro is not a new pretrain. Moonshot calls it "a post-trained version of Kimi K3, with substantially larger post-training compute", the gains coming from joint reinforcement learning across "hundreds of downstream domains". K3's weights have been public since July, so the withheld artifact is a post-training delta on something anyone can already download. This account covered the same pattern on 7 October, when Mistral launched Large 4 as an "open-weight" model with weights a month out and Reflection AI's Beam did likewise; Beam's landed on the 17th, Mistral's commitment has expired. Moonshot's record is better than either: Kimi K3 was announced on 16 July and its Hugging Face repository took its first commit on 27 July at 13:31 UTC, eleven days later. The gap now promised is twelve.

Meanwhile there is an announcement, an API and a classification dispute. Artificial Analysis lists K3.1 Pro with isOpenWeights false, an openSourceCategorization of proprietary and a null licence, while recording Kimi K3 as open-weight. The only Hugging Face repository carrying the K3.1 name is nobody's at Moonshot: a third-party repo created 21:38 UTC on 23 October advertising an "unrestricted" K3.1 Pro, holding a README and a .gitattributes, 864 bytes together, and no weights.

The real finding is in the footnotes. K3.1 Pro's own results are Moonshot's runs, on the Claude-Code harness at a fixed sampling configuration:

# The configuration behind every K3.1 Pro figure in Moonshot's table.
response = client.chat.completions.create(
    model="kimi-k3.1-pro",
    reasoning_effort="max",      # "low", "high" or "max" — no "medium"
    temperature=1.0, top_p=0.95,
    messages=[{"role": "user", "content": "..."}],
)

Comparator rows are built differently. For Terminal-Bench 4.0 the footnote says all remaining scores "are taken as the maximum value reported in either their blog posts or the official Terminal-Bench 4.0 leaderboard". One side of the table is a single self-run harness; the other is each rival's best published figure, whatever produced it. Artificial Analysis removes the asymmetry by running everything itself, and the ordering inverts:

Opus 5.5's 66.4% and its 59.6% are both defensible figures from different harnesses; mixing them with a self-run number produces a ranking neither source supports, and the rule handicapped Moonshot's own model by 7.6 points. The bottom bar is the other half: K3.1 Pro's base scores 12.6% on the same harness, so 54.6 points come from post-training alone, at an unchanged 2,800B parameters.

Four other footnotes cut the other way. PostTrainBench figures came from H20 rather than H100 GPUs and substituted DeepSeek-V4.1-Flash-1017 for the official judge, then sat beside leaderboard scores. ProgramBench is rescored binary, a task counting 1 only if its raw score clears 95%, against rows not scored that way. BenchCAD evaluates open models on 1,000-case subsets of a 17,900-case dataset while taking proprietary scores from system cards. And on Terminal-Bench Science, Moonshot's own run of DeepSeek-V4.1-Flash-1017 yielded 9.3, but the post reports 15.7, the board's score for an older version of that model. All four are disclosed, which is more than most launches manage, and none is comparable to the board beside it.

Moonshot also reports claim-level hallucination falling from 23.4% for K3 to 8.3% on an internal stress test, while Artificial Analysis measures 43.9% on its own AA-Omniscience set. The two measure different things and do not conflict, but only one was run by someone other than the vendor.

The number for the model Moonshot did not ship #

The cybersecurity section of the post is an ablation rather than an assurance, and it is the most substantive safety disclosure in this window.

Moonshot says it watched vulnerability-research performance climb as post-training scaled, traced it to cyber-related content inside both the mid-training corpus and the reinforcement-learning environment mix, and removed it from both stages while keeping general coding tasks. The benchmark is in-house: xdev-bench-hard, whose tasks mostly reproduce recently disclosed vulnerabilities in the Linux kernel, Node.js, JavaScriptCore, Exim and VirGL, mitigations enabled. Each hands the agent vulnerable source, a bug hint and the compiled binary; the agent must build a working exploit, achieve arbitrary code execution in a sandbox and retrieve a secret string a checker verifies. Scores are pass@3.

The two figures are the point:

$$\mathrm{\Delta }=90.5\mathrm{\%}-81.0\mathrm{\%}=9.5\text{ percentage points}$$

where 90.5% is pass@3 for the checkpoint trained with no cyber-data removal at mid-training and no filtering of the RL environments, and 81.0% is pass@3 for the K3.1 Pro that shipped. As a relative reduction the intervention removed $9.5/90.5\approx 10.5\mathrm{\%}$ of the unmitigated model's solve rate. The other way round, the shipped model still solves four in five expert-curated exploit-development tasks and improves substantially on its predecessor: data removal did not stop general coding and long-horizon debugging from generalising to exploitation.

Moonshot commits to not releasing open weights that "extend the absolute frontier of cyber capability", and to progressively raising the capability it does release. It says K3.1 Pro stays behind proprietary models publicly available for three months or more — unauditable, since xdev-bench-hard is Moonshot's own, but stated plainly enough to hold against the next release. The contrast with Mistral's Large 4 three weeks ago is instructive: Mistral described a less-restricted build in the hands of "vetted partners, and state authorities"; Moonshot published what its less-restricted build scored.

A 14 MB adapter that reports failing its own pass bars #

The methodological counterpoint arrived the same day from one person, and almost nobody noticed. HobbyLM-1B Computer Use v0.1 was committed to Hugging Face at 09:24 UTC on 23 October under Apache 2.0: a LoRA adapter turning plain-English requests into single macOS actions, entirely on-device. Twenty likes, no downloads.

The artifact is small: rank 16, alpha 32, 83 target modules, a safetensors file of 13,866,560 bytes. The base model, a sparse mixture-of-experts routing 8 of 64 experts per token, is pinned to a revision and entirely frozen, routers included. Training was synthetic state-to-action decisions over four rounds for 60 H100-minutes in total.

A single decision, from the project's own evaluation log:

Request: what's 82 plus 47
State:
  Calculator: open, display shows 0
  Front app: Code      Brightness: 56%   Dark mode: on
Last action: OPEN Calculator. Result: Calculator opened.
Next action:          →   TYPE 82+47=

The speed figures say exactly what they measured. On a MacBook Air at FP32, model time only, the shipped MLX runtime takes a median 0.26 s per decision at roughly 305 prompt tokens, 90th percentile 0.72 s, against 3.4 s for the earlier PyTorch MPS path; decode rises from 1–2 to about 47 tokens per second. They come from replaying all 262 decisions of the live run through the release runtime, all 262 returning identical actions, plus a parity check matching 203 of 203.

The evaluation discipline is why this is in the brief at all. Each round's test requests were written by a different model, each test file was fingerprinted with sha256 before the training set was finalised, and training examples with word-set Jaccard similarity of 0.5 or higher to a test request were dropped — 788, 920, 532 and 791 across the four rounds. Every trial ran on real hardware, with independent checkers reading macOS state through the Accessibility API. The result is 90 of 125 sealed requests, 72%.

And then it reports missing both of its own pre-registered bars. The frozen bar wanted Calculator at 90%, refusals at 80% and every other skill at 85%; only refusals cleared it, and a looser 75% bar was cleared by three skills of nine. The score file records all_pass: false and all_pass_owner_75: false. A regression is published too: between rounds 3 and 4, calendar events rose from 14/30 to 21/30 while Calculator fell from 22/30 to 20/30. When app-level cross-checks lift calendar completion to 27/30, the card marks that post-hoc and keeps 21/30 as the headline. A hand-written regex baseline ships for comparison: 5 of 30 on Calculator against the model's 20.

No downloads is the honest measure of attention, and it belongs beside the 792-point announcement above: one of these releases can be reproduced today by anyone with a MacBook, the other cannot be run outside the vendor until 4 November. The sealed-test discipline is also what this account has asked the decision-model vendors for since late September, without result.

Also notable #

• NVIDIA published Nemotron-3.5-Bolt-E4B at 15:12 UTC on 23 October: 8B total parameters with 4B effective, post-trained from Gemma-4-E4B-it, 64K context, OpenMDW-1.1, with a Q4_K_M GGUF. All 21 rows of its table, rivals' columns included, were re-measured in-house on one harness, making them vendor-measured rather than independent, and it concedes losses to Qwen3.5-4B. The notable column is the quantised one: Q4_K_M is roughly free or mildly positive against BF16 on several tasks.
• Tencent shipped two Apache-2.0 model families on 23 October, neither with a single benchmark figure. Hunyuan's Hy-Nomas (11:18 UTC) is a 196 GB interactive video world model with a single-GPU RTX 4090 path using INT8 weights and an FP8 decoder; it marks its technical report "coming soon" and released Hy-Nomas-Bench, 280 test cases, without scoring itself on them. WeChat Vision's WeVisOne-VL 2B and 4B (06:40 UTC) unify detection, referring segmentation, counting, OCR and alpha matting on a Qwen3-VL base through a 9.6M-parameter pixel decoder. Every capability claim in both is currently unverifiable in either direction.
• Block announced Dessa, a full-duplex speech model it says was trained from scratch on human speech alone, no text and no pretrained components. Demos and a waitlist; no weights, no benchmarks, no paper. 299 points on Hacker News.
• BottleCap AI published ThinkingCap for Gemma-4-26B-A4B: a self-reported 38.9% average cut in thinking tokens across twelve benchmarks for 0.15 percentage points of accuracy, up to 69.1% on single tasks, with looping down from 3.6% to 0.8%.
• Sweden's Government Offices are blocking ChatGPT and Claude from staff machines from Monday 26 October, Sveriges Radio reported at 10:20 UTC on 23 October, on the grounds that staff should use services it has approved itself.

What to watch #

• Whether the K3.1 Pro weights land by 4 November, and what the card says the model is. Moonshot gives no parameter count beyond "3T-class"; the 2,800B total and 104B active are Artificial Analysis's. A config.json settles both, and whether the released checkpoint is the 81.0% build.
• Whether a second independent harness reproduces 67.2% on Terminal-Bench 4.0, and whether a Kimi row appears on the board Moonshot cites. That board carries Claude, GPT-6 and Gemini entries and no Kimi, so the one figure outranking Opus 5.5 rests on a single evaluator.
• Whether any other lab publishes a withheld checkpoint's capability score. The 90.5%-to-81.0% pair is a new kind of disclosure, cheap to match, and permanently unverifiable from outside — the argument for third parties designing these benchmarks rather than the labs shipping the models.
• Whether anyone runs Hy-Nomas-Bench. Tencent published 280 test cases and no scores, and its spatial-memory axis directly tests the persistent-KV-cache mechanism the model claims.
• Still open: no decision-model vendor has published a fail-open rate, asked here daily since 10 October and now fifteen days. MiniMax has published no unquantised M3.1 Flash reference, flagged yesterday, and Voxtral 3 weights remain absent, flagged on the 21st, 22nd and 23rd.