AI Brief, 14 September 2026: a science model with three extra tokenizers

Shanghai AI Laboratory put the weights for Intern-S2-397B on Hugging Face at 03:33 UTC on 13 September, under Apache-2.0. That is fifty-seven minutes before yesterday's edition closed, so this account is a day late to it, and it is the largest set of open weights released in the window by a wide margin.

The interesting part is not in the model card. It is in the two configuration files. Intern-S2-397B declares the architecture class Qwen3_5MoeForConditionalGeneration, and every structural hyperparameter in it — 60 layers, 512 experts with 10 active per token, hidden size 4096, head dimension 256, two key-value heads, a 262,144-token position limit, one full-attention layer for every three linear-attention layers — is identical to Qwen3.5-397B-A17B, which Alibaba published in February. Exactly one number differs. The vocabulary grows from 248,320 to 251,392, a difference of 3,072. The repository ships three extra SentencePiece tokenizers, for protein sequences, SMILES chemical strings and nucleic acids, and each holds exactly 1,024 pieces. The whole of the declared difference between the two models, at the level the config file describes, is three scientific alphabets bolted onto a Qwen vocabulary.

That framing is worth holding onto when reading the benchmark table, because the model's own comparison set puts Qwen3.5-397B-A17B in the first rival column, and the gap there is the largest in the release. On Biology-Instructions, a multi-omics sequence task, Intern-S2-397B self-reports 55.71 against 4.49 for the Qwen model whose architecture it shares, and against 13.87 for the strongest general model in the table. Those are enormous margins, and they are the lab's own numbers. They also come with an honest counterweight that the lab left in: on general instruction following the model places last of the eight compared, and on terminal agentics it is second from bottom, more than twenty points behind the leader. This is a specialist, and its own scorecard says so.

Elsewhere in the window, a University of Washington and Meta FAIR paper in this morning's arXiv batch argues that byte-level models overtake tokenized ones under distillation — a claim that holds in its extrapolation to infinite compute and in no number it actually measured. And DeepSeek reversed the model retirement it had scheduled for 04:00 UTC today, which the 11 September edition flagged.

• Intern-S2-397B, Apache-2.0, weights uploaded 13 September 03:33 UTC: 806.9 GB of bf16 tensors, 512 experts with 10 active per token, 262,144-token context, plus native time-series input.
• Its config differs from Qwen3.5-397B-A17B in exactly one field: 3,072 added vocabulary slots, matching three shipped 1,024-piece scientific tokenizers.
• Self-reported 55.71 on Biology-Instructions against 4.49 for that Qwen model; also last of eight on instruction following and 64.04 on Terminal-Bench 2.1 against 84.60 for the leader.
• arXiv 2609.12303 reports byte models beating tokenized ones by 4 to 8 per cent — all of it fitted asymptotes. Measured, the gap over its own token baseline is 0.2 points, at 6.35× the inference FLOPs per token of text.
• DeepSeek will keep serving V4-Pro past today's announced cutoff, "in response to user demand". Its two smaller V4 endpoints were retired as planned.

Intern-S2-397B: what the config file says #

The shipped checkpoint is 806,859,499,264 bytes across 188 safetensors files. At two bytes per parameter in bf16 that is about 403 billion, so the 397 billion in the name rounds down; the remainder sits in a 27-layer vision tower and a multi-token-prediction head. The card states no activated-parameter count, but the routed expert stack is where nearly all the weight lives and its arithmetic is checkable:

$$60\times 512\times 3\times 4096\times 1024=386.5\text{B}$$

Sixty layers, 512 experts each, three matrices per expert (gate, up and down), each $4096\times 1024$ — 386.5 billion parameters, 96% of the model, sitting in experts. With num_experts_per_tok set to 10, a token touches 10 of the 512, which is 7.55 billion of that 386.5 billion. The rest of the per-token cost is attention, the always-on shared expert and the embeddings, which together put activated parameters somewhere near 16 billion — consistent with the A17B in the base architecture's name, though the card never says it.

The other half of the design is where attention is allowed to be quadratic. full_attention_interval is 4, so 15 of the 60 layers run standard grouped-query attention and the other 45 run a gated linear-attention variant whose state is a fixed-size recurrence rather than a growing cache.

That ratio is the whole memory story. Each full-attention layer caches two key-value heads of dimension 256, for keys and for values, at two bytes each: 2,048 bytes per token per layer. Fifteen such layers give 30,720 bytes, so a filled 262,144-token context costs 8.05 GB of KV cache. Had all 60 layers been full attention, the same context would cost 32.2 GB. The saving is exactly the 3:1 ratio, and it is the reason a 262k window is servable at all on this parameter count.

For scale in the other direction: DeepSeek-V4.1-Flash, covered here on 11 September, holds its cache in 890 bytes per token. Intern-S2 spends 34 times that. One lab is chasing serving cost, the other scientific coverage, and the cache budget is where that choice becomes visible.

Every figure in that release is self-reported, evaluated by the lab with OpenCompass, VLMEvalKit and AgentCompass, and shipped as a single image rather than a table. Nobody outside the lab has measured this model. Two results are worth flagging. On SWE-bench-Pro under Mini-SWE-Agent, Intern-S2-397B and GLM-5.2 both score 68.54, identical to two decimal places, which the release does not comment on. And on Terminal-Bench 2.1 under Terminus 2 it scores 64.04 against 84.60 for Claude-Opus-4.8. A model that leads on olympiad-level science by eight points and trails on terminal work by twenty is telling you what it was trained for.

The repository ships an FP8 sibling whose weights went up 36 hours earlier, on 11 September at 15:33 UTC. The deployment_guide.md that the card links for serving instructions is a 15-byte file.

A byte-level model that wins only at infinite compute #

Breaking the Token Ceiling (arXiv 2609.12303, submitted 11 September, announced in this morning's batch) comes from the University of Washington and Meta FAIR, and it goes at the opposite problem: not what to add to a vocabulary, but whether to have one.

The technical contribution is real. Distilling a byte-level student from a token-level teacher is awkward because the two do not share an output space, and prior exact methods need several teacher passes per step. The paper gets an exact conversion in one pass by adding a single <eot> symbol to the byte vocabulary — 261 entries instead of 260 — and inserting it after every BPE token in the training data. The residual mass that a naive prefix-marginalisation silently discards is absorbed by that symbol instead:

# teacher puts 0.5 on "isu", 0.125 on "isk", 0.125 on "is"
# marginalising by prefix drops the token that ENDS at "is":
p_u = 0.5   / (0.5 + 0.125)          # 0.800  <- mass lost
# with <eot> in the byte vocabulary, nothing is dropped:
p_u = 0.5   / (0.5 + 0.125 + 0.125)  # 0.667  <- exact
p_eot = 0.125 / 0.75                 # 0.167, the token that ended

Because a byte vocabulary is small enough to fit entirely in memory, the student sees the teacher's full distribution with no top-k truncation, which is the argument for why bytes and distillation should suit each other.

The headline claim is where care is needed. The abstract reports the byte model surpassing its token counterpart by 4%, Llama-3.2-1B by 6.5% and Gemma 2B by 2.1%. Every one of those is an asymptote from a two-stage curve fit extrapolated to infinite compute, not a measurement. At the largest scale actually trained — 880 billion byte units, 64 H200s per run — the byte model averages 44.6 across six benchmarks against 44.4 for the paper's own token-distilled baseline, a gap of 0.2 points, and loses to Llama-3.2-1B at 45.9 and Gemma 2B at 50.3 on the authors' own re-runs of those models. The 2.1% edge over Gemma 2B also depends on a single learning rate: the fitted asymptote is 52.4 at 4e-3 but 49.7 and 49.8 at 1e-3 and 8e-3, both below Gemma 2B. The abstract reports the best of the three.

The cost side is stated plainly and not resolved. A BPE token is 4.5 bytes on average, 5.5 byte-units once <eot> is inserted, so per token of text the byte model spends 51.92 GFLOPs against 8.18 — 6.35× — and the paper's own section on inference costs says matched-cost comparison is "beyond the scope of this study". No latency or throughput is measured, and no weights, code or data are released. Its authors are candid enough to note in their appendix that the extrapolated asymptote is the least stable quantity they fit.

DeepSeek keeps the model it said it would retire #

The 11 September edition flagged DeepSeek's announced plan to route all deepseek-v4-pro requests to V4.1-Flash from 12:00 Beijing time on 14 September, and said the thing to watch was whether it held. It did not. Past that deadline, the pricing page still lists deepseek-v4-pro serving DeepSeek-V4-Pro-0813, with a footnote reading that "in response to user demand, we have decided to continue providing API services for DeepSeek V4 Pro after September 14, 2026, with the billing method remaining unchanged."

The retirement was not cancelled wholesale. The same page confirms that deepseek-v4-flash and deepseek-v4-flash-vision-exp were retired on schedule and now resolve to V4.1-Flash at the Flash price. So the two smaller endpoints went and the largest one stayed, the opposite of the consolidation the announcement described. V4-Pro remains the expensive option: $0.66 per million input tokens off-peak against $0.15 for Flash, and $1.98 output against $0.60, with a concurrency limit of 500 against 2,500 and no vision support. Users paying 4.4× the input price for a fifth of the throughput allowance were evidently enough of a constituency to reverse a published deprecation.

That matters beyond a price list. The 11 September reading of the original notice was that a lab retiring its largest model in favour of one a third the size was making a claim about where capability now comes from. The reversal withdraws it: Artificial Analysis puts V4.1-Flash at an Intelligence Index of 39.55, and enough paying customers evidently agreed it was not a replacement.

Also notable #

• Artificial Analysis published its own measurements of four MBZUAI checkpoints on 13 September, and they form a deliberate training ladder rather than four products: K2 Horizon 0.9B, 3.7B, 7B and MoVA 36B-A4B score 3.05, 16.16, 21.05 and 25.67 on the Intelligence Index, all flagged as measured rather than estimated, all Apache-2.0, and the 7B card states that intermediate checkpoints are published so capability change can be studied across training. A note for anyone reconstructing that index: it is not the average of the components published beside it. No subset of the eleven non-null evaluations on those pages reproduces the stated totals under equal weighting, and the index runs consistently harsher than a naive mean of its own parts.
• SemiAnalysis argued on 13 September that 4-high HBM stacks beat taller ones for inference, since bandwidth per stack is fixed by HBM4E's 2,048 data I/Os regardless of height, making a shorter stack strictly cheaper per unit bandwidth. It puts the cost premium at 12.1% for 8-high and 26.3% for 12-high against throughput gains of 8% and 10% — the firm's own roofline and BOM models, not measurements, and the cost tables sit past the paywall.
• AllSpark Research's Iris-mini and Iris-pro search agents drew coverage on 13 September, but the weights were uploaded on 2 September. Iris-pro is fine-tuned from Qwen3.5-397B-A17B, making two separate labs in this issue that build on the same backbone.
• ElevenLabs shipped Music v2.5 on 13 September. The Information reported an Anthropic compute lease with Rum Group at $13.7bn over six years; that is paywalled and single-source, and appears on neither company's site.

What to watch #

• Whether anyone measures Intern-S2-397B independently. Every number in the release is the lab's own and ships as an image. The weights are Apache-2.0 and 806 GB, so the barrier is hardware rather than access — and the science benchmarks it leads on are ones no public leaderboard runs.
• Whether Shanghai AI Laboratory states a base model. The config declares the Qwen3.5 architecture and the parameter counts match, but the repository declares no base_model relationship — and the 4.49-to-55.71 result means much more once the starting point is known.
• Whether DeepSeek attaches a new date to V4-Pro. The footnote promises "further notice should there be any changes" and gives no timeline, so the endpoint is now indefinitely alive after a published deprecation lapsed.
• Whether Artificial Analysis states a comparability position across index versions, asked here for a tenth consecutive issue. This week adds a second transparency question to the first: the index cannot be reconstructed from the per-evaluation figures the firm publishes alongside it, so a reader cannot check either how versions relate or how a single version is computed.