AI Brief, 30 October 2026: a trillion open parameters, and DeepSeek's attention underneath them

Mistral published the weights of Mistral Large 4 at 2026-10-29T14:11:16Z. This account covered the launch on 7 October as a trillion-parameter open-weight model with no weights, against a promise that they would "drop end of this month". They have, with two days to spare. It is the loudest release of the window by a wide margin, 1,274 points on Hacker News inside a day, submitted two minutes after the commit landed. The artifact also settles the parameter-count discrepancy flagged on the 7th, when Mistral's own two pages disagreed: it is 49 billion active per token and 52 billion counting embeddings and output layers, so both were right about different things. Mistral says it was trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in its own European datacentres.

Three things about the artifact matter more than the parameter count. The checkpoint is FP8 only: 54 shards, 1,060.6 GB, no bf16 original, which puts it where MiniMax's M3.1 Flash was on the 23rd. It arrives under a new licence, the Ecosystem Open-Weight License, a modified MIT granting commercial use but requiring a separate agreement from anyone operating the model as a service. And its two efficiency mechanisms are not Mistral's: llama.cpp shipped support eleven hours later under a commit titled "extend DSA and sdpark to new model", DSA being DeepSeek Sparse Attention. The people who had to write the kernel classified a European flagship's attention as an existing DeepSeek path.

The benchmark picture is split, and the split is the most useful thing in the release. On coding, Mistral's claim is efficiency rather than ceiling: 48% on the Artificial Analysis Coding Agent Index, effectively tied with GLM-5.3 while using 47% fewer tokens per task, and 41% on Terminal-Bench 4.0, up 13 points from the preview with 30% fewer output tokens. On the aggregate index, the one that led this brief yesterday, it scores 26, against the 59.43 that put Gemini 4 Carbon first on 28 October. One caution for anyone who read the 10 October edition: that put Mistral Large 4 at 38 on Artificial Analysis's index, and 26 is not a regression. The index moved to v5 on 27 October with two benchmarks added, three removed and three replaced. The model improved and its printed score fell. Mistral's card concedes the standing it does claim: highest among western open-weight models, with the leading Chinese open-weight models scoring higher. All these figures are Artificial Analysis's rather than Mistral's, quoted in Mistral's materials rather than read from Artificial Analysis's own table.

One passage deserves reading twice. Mistral says that after monitoring cybersecurity use in live API traffic since the preview, it sees "a potential for uplift to conduct real attacks" and is modifying its guardrails accordingly. That sentence appears in the post releasing the weights.

• Mistral Large 4, 1T total and 49B active, published 29 October at 14:11 UTC as FP8 only, 1,060.6 GB across 54 shards. No bf16 checkpoint exists; an NVFP4 variant ships alongside.
• The licence is new: a modified MIT permitting commercial use and fine-tuning, with a separate commercial agreement required to run it as a Model-as-a-Service. Mistral closed its Voxtral speech line to API-only on 20 October; nine days later it opened its largest model.
• The 7 October watch item is closed. Weights promised "end of this month" arrived on the 29th. Reflection AI's Beam, promised Apache 2.0 weights in the same window, still has none.
• Coding is the strong result: 48% on Artificial Analysis's Coding Agent Index at 47% fewer tokens than GLM-5.3, and a lead among open weights on Cognition's private FrontierCode 2.0 at 19.7%.
• On Artificial Analysis Intelligence Index v5 it scores 26, against 59.43 for the model that led this brief yesterday. Mistral's card states Chinese open-weight models score higher.
• Mistral reports a 96.6% refusal rate on malicious cyber prompts and, in the same post, a potential for uplift to real attacks. Guardrails can be revised on an API; the 1,060 GB cannot.
• Outside the labs: a world model of 26,000 to 131,000 parameters now runs games on a Raspberry Pi Pico 2 W, with every predicted frame checked against the real rules.

What the trillion parameters actually do per token #

The config answers what the headline figure does not: what gets touched when a token goes through. 72 layers, 128 routed experts with 4 active per token plus 1 shared expert, only the first layer dense. Multiplied out, the feed-forward stack is about 1,038B parameters of which about 41B are active, which is where the 49B comes from once attention is added. The granular MoE is standard now; the attention is where the choices live.

Keys and values are compressed into a latent vector before being cached, the multi-head latent attention pattern DeepSeek introduced. The figures below are from the published config.json. With a latent rank $d_c=512$ and a rotary component of 64 dimensions, the cache holds $512+64=576$ values per layer per token, where ordinary multi-head attention over this model's 64 heads of width 128 would hold

$$n_h(d_k+d_v)=64\times 128+64\times 128=\mathrm{16,384}$$

values for the same token and layer. That is a factor of 28.4. A second mechanism sits on top: kv_source_layers shows that 9 of the 72 layers do not compute their own keys and values at all and read layer 35's instead, so only 63 layers cache anything.

At the context length Mistral advertises, 1,048,576 tokens, the latent cache is 70.9 KB per token and 76.1 GB for one full-length sequence in bf16. Plain multi-head attention on the same shape would need 2.16 TB, more than twice the weights. That is the whole reason a 1M window is claimable at this size.

The indexer is the part worth sitting with: a separate 32-head scoring module with index_topk set to 2,048. Attention is computed over the 2,048 positions it picks rather than over the context, which at the advertised maximum is 0.195% of the window. So "1M context" here means a model that can address a million tokens while reading two thousand of them per layer — a reasonable trade, and a different claim from the one the number implies.

The checkpoint also ships a speculative decoder Mistral calls DSpark: three dense latent-attention layers in bf16 proposing a block of three tokens via a recurrent Markov head. One flag, and Mistral reports up to 1.9x decode throughput with identical outputs:

# Added to the vLLM serve command; outputs are unchanged, decoding is faster.
--speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'

Against that, the part Mistral does not put in a chart. Eight H200s give 1,128 GB, so 1,060.6 GB of weights leaves 67 GB for KV cache, activations and the draft head: 94% of the node spent before a single request, and less than the 76 GB one full-length sequence needs. Both serving examples use a tensor-parallel size of 8, and the card's advice if memory runs short is to lower --max-model-len. On eight B200s at 180 GB the sums work. The model is open; running it at its advertised context is not a one-node proposition.

A licence that opens, and a guardrail that cannot be recalled #

The Ecosystem Open-Weight License is short and worth reading rather than summarising from the adjective. It reads as MIT, with one substantive condition: a licensee running a Model-as-a-Service business — giving a third party access for inference or post-training in a way that lets them control inputs, parameters or training data — needs a separate commercial licence. Internal use is exempt, as is embedding the model in an end-user product.

That is narrower than the Mistral Research License the Large line previously carried, and it is not Apache 2.0: for almost everyone reading this it behaves like MIT, and for anyone building a hosted inference business on it, it does not.

The harder tension is the security one. Mistral reports a 96.6% average refusal rate on the cyber prompts of JailbreakBench, StrongREJECT and AgentHarm, the highest it measures among open models, plus 97.1% attack resistance on B3 against 93.3% at preview. There is also a Cyber Index entry at last, 51%: on 7 October the only cyber figure was an 82% Mistral quoted from a third party about itself, and on the 10th Artificial Analysis had no entry at all.

Mistral also states that live traffic since the preview shows potential for uplift to conduct real attacks, and that the guardrails are being modified. Both can be true. But a guardrail revision reaches the API, not the download. Moonshot on the 24th published the cyber score of the checkpoint it chose not to ship, 90.5% against the shipped 81.0% — a disclosure about a decision already taken. This is a disclosure about one still being adjusted, attached to weights already distributed.

What that looks like in practice was published in the same window. CrowdStrike reported on 29 October that a China-nexus actor it tracks as VAPOR PANDA ran an agentic intrusion framework called ARTEX against more than 200 organisations in at least 14 countries over 11 days from a single controller, beginning 17 October, driven primarily by DeepSeek with a second commercial model it could not identify. Recovered transcripts show planning agents consuming on the order of 1.4 billion input tokens in one intrusion, with a human steering only at decision points, issuing instructions in Chinese timestamped to China Standard Time. The figures are CrowdStrike's own telemetry and are not independently audited. But the mechanism is not hypothetical: the engine was an open-weight model, and no vendor guardrail was in the path.

A world model small enough to check, on a microcontroller #

Two days ago this brief led on Genie 4, which shipped with no resolution, frame rate, parameter count or benchmark, alongside a Helmholtz Munich paper arguing that high task reward does not show a world model has learned its environment's dynamics. The counter-example arrived on 29 October at 20:32 UTC from a single developer, at 165 points on Hacker News.

tinyworldformer runs two networks on a Raspberry Pi Pico 2 W. One is the world model: frame plus action to next frame plus reward, a looped transformer of 26,000 to 131,000 parameters quantised to int8. The other is an actor of 9,000 to 16,000 parameters that plays the dreamed game. On the chip neither touches the real game: the actor only sees frames the world model generated, and the world model only receives actions the actor chose.

The looped transformer is the idea doing the work. Rather than stacking distinct layers, one transformer block with one set of weights is applied one to four times, with the input added back before each pass, and the output after every pass trained against the next frame. Loop count becomes a runtime dial trading latency for accuracy, and weights are reused, so memory does not grow with depth.

Because each game is a few hundred lines of numpy, every predicted frame can be compared cell by cell against the real rules — the measurement Genie 4 does not offer and the Helmholtz paper asked for, reported here per game and per loop count.

Breakout reaches 99.99% at two loops on 26,000 parameters and 4.3M multiply-adds. The repository is careful about why Doom-lite looks worse: its raycast view has 768 cells, and one wrong shaded wall pixel fails the whole frame. Accuracy declining slightly after three loops is the kind of result a fixed-depth model cannot show you at all.

The engineering below is the reason to read it. Every matrix is int8 with one scale per output row, activations int8 with a separate scale per loop index because the loops see different ranges, and quantisation is trained in, so the int8 model is the trained model rather than a conversion of it. Flash and RAM come from the firmware ELF: Pac-Man, the largest, needs 600 KB and 460 KB against the board's 4 MB and 512 KiB.

One number is missing, and the author says so: the on-chip frame rate has not been measured. The 0.7 to 1.5 ms per frame in the repository is the same Rust code on a desktop CPU, and the 66 to 150 ms quoted for human play is a deliberate throttle, not a ceiling. What is demonstrated is that the firmware runs, fits and is numerically exact; what is not is how fast it runs on the hardware in the title.

Also notable #

• Pika Labs published open weights for the first time on 29 October at 17:10 UTC: not a base model but a LoRA adapter on MiniMax-H3 doing image-to-video with joint audio, under MiniMax's community licence. 121 likes in eleven hours, and no press coverage found.
• GGUF conversions of Mistral Large 4 appeared 4 hours 17 minutes after the weights, from Unsloth at 18:28 UTC — before llama.cpp could load the architecture, which it managed at 01:11 UTC today.
• llama.cpp's Intel GPU path got faster on 29 October: a SYCL flash-attention kernel takes Llama 3.1 8B at 16K context from 31.3 to 54.2 tokens/s on an Arc B580. Build b12000, at 04:25 UTC today, adds the OCP FP8 E4M3 conversion support that FP8-only releases now require.
• SkyPilot argues GB200 racks are scheduled wrongly, and puts a number on it: treating NVL72 racks as gang-schedulable slices rather than 8-GPU nodes yields 2,037 usable nodes against 1,602, a 27% gain from identical hardware. The company's own measurement, 133 points on Hacker News.
• The papers stream produced nothing loud, and the reason is worth knowing: the ~500 arXiv submissions in the window are hours old and carry no votes yet, while every heavily upvoted paper on Hugging Face's Daily Papers page is three to eight days old. Curation date is not publication date.

What to watch #

• Whether anyone independently measures Mistral Large 4's cyber numbers. The 51% Cyber Index figure and the 96.6% refusal rate are both checkable by a third party, and the weights are downloadable, so nothing prevents it.
• Whether Mistral puts a figure on the uplift it reports observing, rather than only on refusals.
• Kimi K3.1 Pro weights, promised by 4 November and flagged here on the 24th: five days left, and no repository under moonshotai yet.
• An on-chip frame rate for tinyworldformer, the one figure between "the firmware fits" and "the board plays at speed", which the author has flagged as unmeasured.
• Whether ARTEX spreads further. CrowdStrike notes this campaign is distinct from ARTEX activity others have reported under Storm-0940, which suggests the framework is circulating among more than one China-nexus group.