benchmarking
Posts tagged “benchmarking”.
-
AI Brief, 14 October 2026: a million-token output budget that arrives capped at 65,536
Gemini 4 Argon became callable on 13 October, two weeks after Google announced it with no general availability. The million-token output budget that led that announcement is not what the first third-party surface exposes. Datalab shipped a 1M-parameter OCR model that reads 49 pages a second on CPU, and an Apache-2.0 reimplementation of Jev claims embodied-reasoning scores with no robot training.
-
AI Brief, 6 October 2026: two tokens that do what reinforcement learning does
A paper published on 5 October lifts Olmo-3-7B on MATH-500 from 42.2% to 77.9% by appending two tokens to the prompt, more than reinforcement learning gained on the same model, then proves the mechanism by editing the training corpus. Reflection AI announced a 501B open-weight model with no weights. And three independent Jev evaluations landed at once.
-
AI Brief, 5 October 2026: a 125B model on a 12 GB card, and a headline nobody measured
Strata took 662 points on Hacker News for running a 125B model on a consumer GPU, but publishes no RTX 4090 measurement at all, and the thread's one careful comparison found a threefold vision regression against plain llama.cpp on identical weights. And the open-weights decision models this brief kept asking about are not missing from JevBench's sealed tier: they are leading it.
-
AI Brief, 2 October 2026: three vendors shipped a decision model, and the sealed scores went negative
Cloudflare, Amazon and Perplexity all published open decision models inside thirty-four hours, and SGLang added a /v1/decisions endpoint. On Cloudflare's own leaderboard, Clef and clef-flash are the only two entries of 73 with no calibration score at all. And JevBench now publishes sealed-tier numbers: the open-weights entrants come out below zero.
-
AI Brief, 1 October 2026: priced, measured, and not yet released
Google announced Gemini 4 Argon on 30 September with a price, a fifteenfold larger output budget and state-of-the-art claims, but no general availability. The one third-party composite already carries it, at fourth. And a paper measures what the decision-model class does with an ordered scale: it stops using most of it.
-
AI Brief, 25 September 2026: the half of the benchmark nobody can see
The independent evaluation of Jev asked for here on the 24th arrived a day later, and it puts a 4B open model first and Jev second. The finding underneath the ranking is larger: on the 308 decisions held back from everyone, the best system in the field scores 36.7% against a 29.3% chance floor. Two typed-decision models launched into that result, one of them ranked 78th of 89 on it.