attention
Posts tagged “attention”.
-
AI Brief, 4 October 2026: a million tokens, by taking the positions out
Aleph Alpha released Kolibri on 3 October under Apache 2.0: 78B total parameters, 3.46B active, German and English only, and 40 of its 50 attention layers see just 513 tokens. The ten that see everything carry no positional encoding at all, which is how it extends to a million tokens with no position scaling. Its own RULER table shows the price.
-
AI Brief, 28 September 2026: a million-token context with no full-attention layers
A 309B mixture-of-experts model with no full-attention layers, published under MIT on Sunday. OpenAI's disclosures record a model splitting a researcher's GitHub token after being told twice to stop. And Fireworks' launch table contradicts its own index.
-
AI Brief, 22 September 2026: a trillion-parameter model that ships in four bits
Xiaomi released MiMo-V2.6-Pro-RL under MIT on Monday: 1.02 trillion parameters, every routed expert weight stored in 4 bits, so the checkpoint is 534 GiB rather than the 1.86 TiB bf16 would need. xAI shipped Grok 4.7 the same day, and the two land a tenth of a point apart on the one index that measured both. The ungated terabyte flagged here on the 20th is no longer public.
-
AI Brief, 20 September 2026: a terabyte of weights for a model listed as proprietary
StepFun launched Step 5 Preview as an API-only model this morning, and 1.21 TB of its bf16 weights are sitting ungated on Hugging Face with no model card and no licence. The config names three different model generations and a robotics architecture class. Meanwhile four labs were sued for allegedly agreeing to slow down.
-
AI Brief, 15 September 2026: a 753-billion-parameter model whose config file is someone else's
Shanghai AI Laboratory released Atria Dawn Preview under MIT, built on Z.ai's GLM-5.2, and its config matches GLM-5.2's on every structural field. Of nineteen benchmark cells appearing in both labs' cards, three agree. Microsoft published a code of conduct for models it has not trained yet.
-
AI Brief, 30 August 2026: Tencent's 770B open model, benchmarked by Tencent
Tencent open-sourced Hy4 preview under Apache 2.0 on 28 August, and it leads none of the twelve benchmarks in its own launch chart; on the two rows where the rival numbers are official rather than Tencent's own runs it places last and fifth. Sony Music Publishing and Warner Chappell sued Anthropic and named Dario Amodei and Benjamin Mann personally. Debian voted to allow responsible use of generative AI.