AI Brief, 14 November 2026: the cache that gets faster as its hit rate falls

llama.cpp released v1.0.0 at 10:20 UTC on 13 November, three and a half years after the project began, and the headline is a promise rather than a feature: from now on llama.cpp and ggml will signal breaks to the public C API with a major-version bump. For a library that much of local inference is built on, a stated compatibility boundary is worth more than most model releases. It arrives with a bill: llama_set_warmup, 35 other deprecated functions and the --defrag-thold flag are gone.

The most interesting thing in it is a cache that runs against the intuition caches usually carry. Mixture-of-experts models no longer have to fit their expert weights in GPU memory: v1.0.0 is the first stable build with a host-memory expert cache, exposing two knobs, how large the cache is and what share of misses run on the CPU rather than being fetched to the GPU. On the contributor's own desktop, with a 6.3 GiB cache, moving misses to the CPU took decode from 24.2 to 47.6 tokens a second while the hit rate fell from 73.3% to 61.5%. At a 19 GiB cache the same change made things worse, 65.5 down to 54.1. Hit rate is the wrong thing to tune for, and the release ships a script that searches both parameters.

The window's other substantial publication is dated 12 November and was not picked up here at the time. Perplexity published its training and evaluation account for Decider v1.3 and Decider Flash v1.3, decision models whose weights sit on Hugging Face under Apache 2.0 and, contrary to the instructions on their own model card, neither private nor gated. That matters for a question this account has put to every vendor in the class since 10 October, now day 36: what fraction of production requests fall below a usable confidence threshold, and how often the ones above it are wrong. Perplexity does not answer it either. But it is the first in the class to ship weights, and llama.cpp has carried decision-model support since 2 October, so the measurement no longer needs a vendor.

• llama.cpp v1.0.0, 13 November, 10:20 UTC: semantic versioning now binding, 35 deprecated llama.h functions removed, FP8 KV caches via -ctk/-ctv f8_e4m3, and architecture support for Ripple-65B-Preview.
• The new expert cache's two parameters interact non-monotonically: on the figures published with it, the fastest configuration at a small cache is the one with the lower hit rate.
• Perplexity Decider v1.3, post dated 12 November, weights uploaded 9 November: Apache 2.0, 48.6 GiB for the 27B, 9.6 GB for the 4B Flash, and a new verbatim span-extraction mode.
• Its own documentation puts 1.5 to 2.2% of top answers at stake on a single constructor argument, against the 0.27% flip rate it advertises as its robustness result.
• Nothing new reached Hugging Face from any of the twelve labs swept for exact upload times. Voxtral 3 weights remain absent, day 25.

llama.cpp 1.0, and an expert cache that rewards missing #

The v1.0.0 release landed at 10:20:19 UTC on 13 November, with ggml v1.0.0 an hour and three quarters earlier, and an accompanying post from Daniel Bevenius, Georgi Gerganov and merve setting out the rule: a breaking change to the public C API is a major bump, a feature or new model support a minor one, a bug fix a patch. One removal will bite quietly, since --defrag-thold and the defrag_thold context parameter are both gone, so a serving script that tuned KV-cache defragmentation now fails to parse its own arguments. The server side gained hardening rather than features, including HF_TOKEN restricted to huggingface.co hosts, the same class of mistake as the credential leakage covered here on 8 November.

The expert cache is the substantive change. In a mixture-of-experts model only a few experts run per token, so the expert set need not be resident; it needs to arrive before the GPU stalls. v1.0.0 adds moe_cache_size and moe_cache_cpu_share to llama_context_params, exposed as --moe-cache-mib and --moe-cache-cpu-share, with an LRFU replacement policy and no CPU wait on cached experts. A hit is served from GPU memory; a miss is either uploaded over the bus or run on the CPU, and the share decides which.

The reason the two knobs fight each other is in the machine's own numbers. Expert uploads to the GPU ran at 9.6 GiB/s while the CPU read and executed experts at 48.7 GiB/s, and each gibibyte the CPU handled removed about 19.9 ms of wall clock from the upload path, because the two overlap. A miss is therefore not uniformly expensive: one the CPU absorbs is roughly five times cheaper to feed than one the bus must carry. With a cache too small for the working set, raising the CPU share turns expensive misses into cheap ones and throughput rises as the hit rate drops. With a cache large enough that misses are rare, the CPU is slower than a hit and the setting costs you.

Worked through on the model those runs used, a 4-bit Qwen3.8-Flash-Next with 64,800 MiB of experts in host memory: a 24,000 MiB cache holds 37% of the expert set and reaches a predicted 83.0 t/s at CPU share zero, against 15.5 t/s from a 3,200 MiB cache. At 3,200 MiB the best share is 0.98; at 24,000 MiB it is 0. The crossover belongs to the machine, not the model, which is why the release ships a search:

# Searches both parameters on the machine that will serve the model.
python3 scripts/moe-cache-tune.py -- \
  build/bin/llama-server -m model.gguf -cmoe -t 8 -c 32768
# -> Recommended: --moe-cache-mib 24000 --moe-cache-cpu-share 0, 83.0 t/s

One of the three new architectures closes a loop: Ripple-65B-Preview, listed here for weeks as a model with no independent number attached to it, is now supported, so anyone can run it and produce the first.

A decision model with downloadable weights, and the flag that moves its answers #

The post is dated 12 November, a day after this account covered Google's entry into the same class, and should have been picked up then. Perplexity released Decider v1.3 on Qwen3.8-27B and Decider Flash v1.3, a distillation of it on Qwen3.5-4B, with weights uploaded on 9 November at 08:38 UTC. The manifest's own byte counts come to 48.59 GiB for the larger model, matching the card's stated 49 GiB requirement, and 9.63 GB for Flash. The card says the repository is private and to authenticate with a token; it is not, and a weight shard downloads anonymously. Perplexity credits Jev, TypeSafe AI's September launch, as its inspiration, the clearest statement yet that this is one model class rather than four products.

The architecture is more legible than Google's because the files are public. Full attention is noncausal over the sequence, so there is no token-by-token decode, and the readout is a BF16 [255, hidden_size] head applied at a marker token, one per question, a pruned slice of the base model's vocabulary projection rather than an lm_head. Probabilities are a softmax over a question's valid options only, divided by a single temperature fitted on held-out data and stored in the repository as 1.2263882. Worth noting against the calibration claims that led the Google item three days ago: decision_config.json records brier_weight: 0.0 and cross_entropy_weight: 1.0, so nothing in the training objective optimises a proper scoring rule. The calibration is entirely post-hoc, and the card concedes a single temperature cannot be right for every task.

The sharper finding is which configuration the published numbers describe. The post says the team evaluated three attention layouts and "ultimately chose isolated questions", where the state attends to all questions and each question attends only to the state and itself. The shipped default is a fourth arrangement, separate, giving each question its own sequence and re-reading the state each time. The card is explicit that separate is "the computation all the scores above were measured with", and that switching to the training-time isolated layout changes the top answer on 1.5 to 2.2% of samples. The fast path moves a further 0.3 to 0.6%.

Set that against the robustness result the post leads with. On 372 multiple-choice questions and 4,788 requests per model, reordering the options, reassigning labels and renaming them changed Decider v1.3's semantic answer on 0.27% of questions, against 2.69 to 4.84% for Jev 1.14; repeated identical inputs changed nothing, while Jev 1.14 moved on 1.88%. That is a real stability advantage over the incumbent. It is also smaller, by roughly an order of magnitude, than the swing available from a constructor argument in the vendor's own library. The two were measured on different samples and are not strictly comparable, but the ordering is the point: the configuration you serve in moves more answers than the perturbation the robustness test applies.

from pplx_decider import Decider

# The benchmark numbers describe this default; `isolated` is the training-time layout.
decider = Decider.from_pretrained(ckpt, device="cuda", mode="separate")
r = decider.decide({
    "state": "Integration failing with a 401 since we rotated our API key.",
    "questions": {"urgent": {"type": "noul", "instructions": "Is the customer blocked?"}},
})
print(r["answers"]["urgent"]["noul"])   # P(true); there is no option below which it declines

On abstention, the position is unchanged from the 12th: a decision question always returns a distribution and a confidence, and nothing declines. The exception is the new locate mode, which extracts verbatim spans and can return nothing, because its span head predicts a per-field threshold trained with an InfoNCE variant whose second term pushes that threshold above every wrong span. The ability to say "nothing here" was built for extraction, not extended to decisions.

Everything quantitative in the post is self-reported and mostly internal: 91.6% mean accuracy over 63 query classification tasks, 6.54 points above Perplexity's own production classifiers, and 93.98% agreement with frontier-LLM judges over 2,318,887 questions sampled from a million production requests. The three external leaderboards named are cited without placements, and agreement with an LLM judge is not correctness.

The class also gained library support in the window: transformers v5.23.0, at 14:11 UTC on 13 November, adds a JevForDecision architecture, nine weeks after Jev launched.

NVIDIA, with no reason given #

NVIDIA's leadership change is the business story of the window and has not been covered here. The company's 8-K, filed 12 November, says the board "accepted the resignation of Jen-Hsun Huang, as NVIDIA's President, Chief Executive Officer and member of the Board, effective immediately" on 11 November, made CFO Colette M. Kress interim chief executive, and appointed Harvey C. Jones, a director since 1993, as first-ever board chair. Neither the filing nor the press release states a reason, and a separate 10 November statement from the independent directors says outside counsel has been retained for an investigation.

What falls inside this window is reaction, thinner than its volume suggests. Reporting on 13 November describes an internal petition for Huang's return, but no petition document, date or signature count could be traced past that secondary coverage, and the only quantified item is a self-selected anonymous workplace survey, which is not a petition. A paywalled Wall Street Journal account the same day alleges NVIDIA funded AI startups that then bought its GPUs; that is secondhand and its figures are not reproduced here.

Also notable #

• OpenAI launched ChatGPT Wallet, an agentic checkout product, at 15:00 UTC on 13 November. Two security products named Daybreak appear in trade coverage the same afternoon but not in OpenAI's feed, so they are unconfirmed.
• transformers v5.23.0 also adds Inkling 2, the 594B-total, 17B-active mixture of experts routing 8 of 384, whose Apache-2.0 weights were covered here yesterday, which is what makes the 500 GB floor noted then testable. trl v1.18.0, 15:37 UTC, makes AsyncGRPOTrainer stable and removes its trl.experimental imports, breaking existing scripts.
• The first evaluation of decision models by someone who does not sell one appeared on arXiv on 12 November. Actor, Judge, or Teacher? finds all six it tests fail as agents acting in an environment, while JEV as a reinforcement-learning teacher lifts a 1.5B policy from 72.8 to 82.9% on ALFWorld and 56.8 to 71.4% on WebShop.
• Anthropic's API and apps failed authentication between 09:58 and 11:58 UTC on 13 November, per its own incident record, with a fix at 11:05 and no cause stated.
• inclusionAI's Ling 3.2 Flash was listed on OpenRouter at 17:45 UTC on 13 November, a hybrid-reasoning multimodal model with a one-million-token context at zero listed price. No weights exist under that name on Hugging Face and no vendor announcement could be found, so provenance and licence are unverified.
• Trade coverage on 13 November presented Google's tpu-sync, an Apache-2.0 KV-cache offload library for Cloud TPU inference, as a launch. The repository dates from 9 May and has never cut a release.
• Gödel's workbooks, read by a model. Since 2017 the logician Jan von Plato has led a Helsinki University project on 16 notebooks Gödel wrote in the early 1940s, 1,009 pages largely in Gabelsberger, a shorthand obsolete even when he used it. Progress ran at about a page a week until September, when von Plato enlisted Claude and, the Guardian reported on 13 November, got a digest of the whole corpus within days. That the pages hold hundreds of proved theorems is so far von Plato's own claim, with no paper or transcription released to check it against.

What to watch #

• Whether anyone computes a decision model's fail-open rate now that nothing stands in the way. Day 36 of asking, and the first day the question does not need a vendor: Apache-2.0 weights, a GGUF of the small model with eleven thousand downloads, and a runtime supporting the architecture since 2 October. The shape of the answer is unchanged from the 12th: at a stated threshold, what fraction of requests fall below it, and what is the error rate above it.
• Whether Perplexity's benchmark table gets restated in the configuration the post says it chose. Every number was measured in separate, and the design described is isolated. The cheap check is a rerun of the 372-question robustness set under both.
• Whether the semantic-versioning promise survives its first awkward case. Removing 35 functions is the easy direction; the test is the first wanted feature needing an ABI break.
• Whether Ripple-65B-Preview gets a number from someone who did not train it, now that a stable runtime supports it and the measurement costs an afternoon.
• Still open: Voxtral 3 weights, flagged daily since 21 October, day 25 with nothing under mistralai since 15 October. No independent number for Tencent's DSpine drafter or Nanbeige4.3-33B; no placebo control for the weight-space programming result; no assessment from the three complexity theorists who agreed on 26 October to examine the Exponential Time Hypothesis manuscript; no account from Cursor of how the tampered extension reached its marketplace; and no response from any company named in yesterday's Blackwell checkpoint-forensics analysis.