AI Brief, 15 November 2026: can a decision model say none of these?

For thirty-seven days this brief has asked decision-model vendors one question and been answered by none of them: when the model is unsure, how often does it wave something through anyway? It has now been answered from outside, by people who sell nothing. SO-Arena, from a group at the National University of Singapore, scores 21 systems on 2,919 tasks and measures two things ordinary leaderboards omit: whether a model keeps its answer when the options are shuffled, and whether it can tell the correct option is missing. The paper went up on 11 November and has not been covered here until now; a row for Google's Gemini Decision 1, covered here on the 12th, was added on 13 November.

The second measure matters most, and it is not an accuracy score. The authors show each question twice, once intact and once with the correct option deleted, and ask whether the model's own confidence separates the two cases. Scored as a ranking, zero is chance. Across 19 systems the results run from 0.79 down to 0.014, and track accuracy only loosely: the best performer on capability ranks seventh of nineteen on reliability, and the two most reliable rank eighth and sixth on capability. A vendor's accuracy number tells a buyer almost nothing about whether the model knows when to stop.

The worst result belongs to a model this brief has cited as its own most instructive miss. Laya, the Apache-2.0 release from Convai Innovations that topped Hugging Face trending in September with more than eighteen thousand GitHub stars, scores 0.014: indistinguishable from guessing. That is a stronger finding than a low accuracy score, because accuracy can be fixed by threshold-tuning and this cannot. If the signal separating "the answer is here" from "the answer is absent" is not in the model's output, no threshold recovers it. Popularity measured that the model was downloaded, not that it worked.

Elsewhere, a Hamburg security firm scanned 159,041 applications built on the AI app-builder Lovable and found 7,403 returning database rows, including passport numbers, to unauthenticated requests.

• SO-Arena scores 21 decision models on 2,919 tasks; its leaderboard uses the hardest tier of each of 11 domains. Code and data are public.
• Missing-answer detection ranges from 0.793 (JevAny-27B) to 0.014 (Laya). Option-order stability is high for most: 14 of 18 above 0.8.
• Gemini Decision 1, added 13 November, scores 0.652 on missing-answer detection and 0.863 on stability. OpenAI Decisions is the least stable major API at 0.678.
• Perplexity's Decider was tested at v1.2, one version behind the v1.3 weights covered here yesterday.
• A scan of 159,041 Lovable apps found 24,664 returning database records without a login, and email addresses in 10,560 tables across 7,403 sites. The authors caution 30% is not a confirmed leak rate.
• DeepSeek V3's architecture was reimplemented in 190 lines of JAX and trained on a laptop, with an ablation isolating what the load-balancing bias does.

The benchmark that measures whether a model can say "none of these" #

SO-Arena is by Lichen Li, Ziping Dong, Sicheng Feng, Qi Li and Xinchao Wang, submitted on 11 November under CC BY 4.0. It covers 11 domains in five capability families, 2,919 tasks, of which 1,232 form the leaderboard: the hardest tier of each domain, every domain weighted equally. Tasks include web and tool sandboxes, and reference labels were manually verified against what the model actually sees.

The reliability half is the contribution. Both components are chance-corrected and defined against perturbations, not a held-out set.

Option-order stability $S$ asks whether the answer survives shuffling. Each item is re-asked under 12 reorderings, and $A$ is the fraction whose answer matches the original. Because a model guessing among $K_i$ options agrees with itself by luck, $A$ is corrected by the chance rate $p_e$ , the mean of $1/K_i$ :

$$S=\frac{A-p_e}{1-p_e}$$

Null discrimination $N$ asks whether the model notices an absent answer. Every item is shown twice, intact and with its correct option removed, and the model's implied probability that no option fits is used to rank the two copies. With $\mathrm{AUC}$ the probability that a copy missing its answer is ranked above an intact one:

$$N=2\mathrm{AUC}-1$$

so $N=0$ is chance and $N=1$ perfect separation. The two combine by geometric mean, scoring zero unless both beat chance:

$$R=\sqrt{S\cdot N}$$

Laya is the worked case for why the geometric mean is the right choice. Its stability is 0.354 and its null discrimination 0.0139, so

$$R=\sqrt{0.354\times 0.0139}=\sqrt{0.004921}=0.0701$$

reported as 7.02 out of 100. Hold stability fixed and raise null discrimination to a mediocre 0.5 and the score becomes $\sqrt{0.177}=0.421$ , six times higher. A near-zero on either axis cannot be bought back with a strong score on the other, which is the right behaviour for a metric describing whether a decision can be trusted.

Three results set against earlier coverage. Gemini Decision 1, whose documentation this brief read on the 12th as stating the model cannot decline to answer, scores 0.652 on exactly the capacity that documentation disclaims, and 0.863 on stability: ninth of nineteen. OpenAI Decisions has the weakest order stability of any major API at 0.678. And Perplexity's Decider was measured at v1.2, one release behind the Apache-2.0 v1.3 weights covered here yesterday, so its 0.613 and 0.508 describe the previous checkpoint.

Two caveats belong on the record. Jeva-4B comes from the authors' own institution and places seventeenth on reliability, which is not the direction self-interest would push; its stability of 1.000 is an artefact of its option cache, which the authors flag themselves. The benchmark's code and data are published, which is what makes the rest checkable. Every open-weight system answered in under 0.3 seconds per task on one H100.

159,041 apps, and the 7,403 that answered anyone #

The Hamburg security firm Crissy Field, working with Handelsblatt, used Common Crawl to identify 159,041 applications built on Lovable between December 2025 and September 2026; 34,545 used Supabase. 24,664 returned database rows as soon as the page loaded, with no login. That alone is not a fault; the data may be public by design. Further read-only requests with the key the page itself ships found email addresses in 10,560 tables across 7,403 of those sites, 30%.

The specific exposures are hard to argue with: one German lifestyle forum served more than 80,000 profiles with names, email addresses and dates of birth plus more than 200,000 chat messages; one service provider exposed more than 1,000 records containing passport numbers, expiry dates and nationalities; another had 280,000 customer emails readable. Tens of thousands of contracts, invoices and ticket records, photo albums and internal messaging also came back.

The mechanism is not an AI failure in any interesting sense, which is what makes it serious. A Supabase client key is designed to be public; what protects the data is row-level security configured on the database. Generate the app without those policies and the result is a working product that also answers every question a stranger asks it. The researchers' framing is the right one: this happens in hand-written code too, but a generator produces it in a recurring pattern, so an attack that works against one app works against the next for free.

Their limitations section is unusually honest: email addresses may be public contact details or test data, the method misses personal records containing none, the 30% "should therefore not be read as a confirmed data leak rate", and a Common Crawl sample is not representative. Only parts of each dataset were retrieved, so affected tables may hold more. Handelsblatt put the findings to Lovable; no response is recorded, and none has appeared since.

The dating deserves a note, because the reception is a lesson in itself. The report updates a Handelsblatt article of 2 November and carries no date of its own. The identical URL was posted to Hacker News on 2 and 3 November, drawing 2 points and then 5. Reposted on 14 November at 17:43 UTC under the blunter headline "Lovable apps expose passport details and more than 200K private messages", it took 711 points and 291 comments. The research did not change. The sentence describing it did.

DeepSeek V3 in 190 lines, and what the bias term is for #

Vaibhaav Tiwari published DeepSeek V3's architecture in 190 lines of JAX under MIT, posting it to Hacker News on 14 November at 15:04 UTC, where it drew 203 points; the repository was created on 13 November and had 138 stars by the next morning. It implements multi-head latent attention, DeepSeekMoE, auxiliary-loss-free load balancing and multi-token prediction, and it trains: 36.2M parameters with 13.6M active per token, 12,000 steps over roughly 98M tokens of 207 books in about 2.6 hours on a 24 GB MacBook Air, validation loss falling from 5.02 to 3.51. The author is careful that it is "an architecture demo, not a strong language model".

What earns it space over the other weekend reimplementations is one ablation. DeepSeek V3 uses no auxiliary loss for balancing; it adds a per-expert bias to the routing logits and nudges that bias on recent load. Turning it off at step 2000 moves per-expert utilisation from a 5.7–6.9% band to 0.09–18.6%, a roughly 200-fold spread between least and most used expert, while the loss is 3.921 against 3.923.

# scores: [tokens, n_experts] router logits. bias: [n_experts], updated per step.
top = jax.lax.top_k(scores + bias, k)[1]            # bias steers routing only
gate = jnp.take_along_axis(jax.nn.softmax(scores), top, -1)  # gradients see raw scores
load = jnp.bincount(top.ravel(), length=n_experts) / top.size
bias = bias + gamma * jnp.sign(load.mean() - load)  # overloaded -> bias down

Together the two numbers say that at this scale balancing buys essentially no loss. It is not a quality mechanism but a utilisation one, and it only pays once experts are sharded across devices and the slowest device sets the step time: an expert taking 18.6% of tokens instead of 6.2% is a three-fold stall on its shard, invisible in a single-device loss curve. That is why the bias steers the routing decision but stays out of the gradient path, as the scores versus scores + bias split above shows.

Also notable #

• The NVIDIA story moved from filings to campaigning. Altimeter's Brad Gerstner and Atreides's Gavin Baker published a letter on 14 November urging the board to consider Huang's return, disclosing a combined stake of 2.13 billion dollars; a public petition passed 100,000 signatures the same day. Analysts quoted by Tom's Hardware attribute the departure instead to export-control evasion, claiming more than 1.5 GW of banned accelerators reached China; that is secondhand and unaudited, supported by no filing or regulator statement. Nathan Lambert's Interconnects argues the concrete consequence is to US open-weight funding.
• Frontier-scale serving on two desk machines, measured. NVIDIA's write-up of 13 November, uncovered here until now, serves the 753B-parameter GLM-5.3 NVFP4 checkpoint, about 464 GB, across two GB300 DGX Stations at tensor parallelism 2. Throughput peaks at 607.8 tok/s at concurrency 16, then falls to 385.0 at concurrency 32, where mean latency rises from 136 to 454 seconds and per-user throughput collapses from 115.3 to 13.0 tok/s. The figures are the post's own with no baseline, so they establish feasibility rather than efficiency.
• Weight-space programming shipped as a tool, and its weakest result is the useful one. Cmpnd released dspy-wsp, packaging the weight-editing result as a DSPy optimizer. Its own table reports Gemma 3 270M moving from 6.8% to between 7.0 and 7.7%, close to no effect. The placebo control asked for here on 3 November is still missing, but this is the first published negative.
• Gemini 4 Carbon tops MathArena at roughly fifteen times the cost of GPT-6.1 Sol, per the leaderboard's own run. The footnote that doubles Carbon's rate on 30 November, flagged here on the 11th, has not taken effect, so the ratio uses the current rate.
• Trade coverage on 14 November presented Runway's Praxis-1 robot-control model as newly open-sourced. Its Hugging Face repositories date from 3 to 12 November and the code from 26 October, so the release is not new, and no licence is declared in the model metadata.
• An independent reproduction of JEPA world-model planning reports 68% success on the original 50 episodes against 44% on 250 fresh ones, a seed change alone flipping 38 of 250. No code, weights or data accompany it, so it stays a claim rather than a result, and nobody else can act on it, until the author ships the material.

What to watch #

• Whether anyone reruns SO-Arena on Decider v1.3. The cheapest outstanding measurement in the field: the evaluator and reliability runner are published, the data is on Hugging Face, and the v1.3 weights are Apache-2.0. The figure worth having is null discrimination, since the 0.613 on the table describes v1.2.
• Whether a vendor publishes a threshold-conditioned fail-open rate, day 37. SO-Arena changes the terms without answering it. No vendor can now call the property unmeasurable, having had it measured on its own model by strangers; what is still unpublished is behaviour on real traffic at a chosen threshold.
• Whether Lovable responds, and whether the response is a platform change. The fix that would matter is a default, not advice: row-level security on for generated tables, or a refusal to deploy a table the anonymous key can read. A documentation page would say the opposite.
• Still open: Voxtral 3 weights, flagged daily since 21 October, day 26 with nothing under mistralai since 15 October. No independent number for Tencent's DSpine drafter or Nanbeige4.3-33B; no Edge0 ablation separating quantisation from post-training, asked on the 13th; no assessment from the three complexity theorists who agreed on 26 October to examine the Exponential Time Hypothesis manuscript; and no response from any company named in the Blackwell checkpoint-forensics analysis of the 13th.