The leaderboard story this autumn was about frontier labs trading points at the top. The more interesting story was further down: a brand-new category of decision models that don't generate text at all, a diffusion LLM running at over 1,000 tokens per second, a post-train that makes Kimi K3 think 35–71% less, and three serious open-weight models from outside China. Here are the nine I bookmarked between early September and early October, what each one is actually for, and where the vendor claims end.
For where these sit against the overall frontier, see the companion LLM Landscape leaderboard and the model selector. Unless stated otherwise, benchmark numbers below are vendor-reported; independent Artificial Analysis (AA) scores are called out explicitly.At a glance
| Model | Vendor | Released | Category | Size | Weights | Price ($/M in / out) |
|---|
| Mercury 2.5 | Inception | Sep 8 | Diffusion LLM | undisclosed | Closed | $0.20 / $0.75 (promo $0.04 / $0.15) |
| Laya | ConvAI Innovations | Sep | Decision model (encoder) | 322M–421M | Apache 2.0 | Self-host |
| Ember-1 | Fireworks AI | Sep 23 | Efficient reasoning (Kimi K3 post-train) | K3-based | Closed (preview) | ~$3 / $15 (aggregators) |
| Strands Decider 2B | Strands Agents | Oct 1 | Decision model | 2B | Apache 2.0 | Self-host |
| Clef / Clef-flash | Cloudflare | Oct 1 | Decision model (text + vision) | 27B / ~9B | Apache 2.0 | $0.24 / $0.038 per M input |
| Kolibri-1 | Aleph Alpha | Oct 3 | Sovereign EN/DE MoE | 78.1B (3.46B active) | Apache 2.0 | Enterprise |
| Beam | Reflection AI | Oct 5 | Open frontier MoE | 501B (23B active) | Apache 2.0 | Waitlist API |
| Mistral Large 4 | Mistral AI | Oct 6 (weights Oct 29) | Open frontier MoE, multimodal | ~1T (52B active) | Mistral open-weight license | $0.70 / $2.10 |
| d1-3B / d1-omni-600M | Liquid AI | Oct 7 | Decision model (edge) | 3B / 600M | Open weights | Self-host |
1. Decision models: a category is born
Five of the nine releases above are decision models. They take a question plus a fixed set of allowed answers — pick one of these labels, score this from 1–5, yes or no, which tool should I call — and return a calibrated probability for each option in a single forward pass. No tokens are generated, so there is nothing to parse, nothing to hallucinate outside the schema, and latency is measured in milliseconds rather than seconds.The incumbent they are all chasing is TypeSafe's closed, API-only Jev. Nearly every launch post benchmarks against it, and on October 13 Artificial Analysis launched a dedicated Decision Index (Jev 1.13 scored 58 at launch). Watch out for a naming collision: there is also a community-run "Jev Decision Index" on Hugging Face. The two use different scales, so don't compare scores across them.Why this matters in practice: a large share of "agent" traffic is not open-ended generation. It is routing, classification, guardrails, tool selection, and extraction-to-enum. Sending those through a frontier chat model is slow, expensive, and hard to calibrate. A decision model is the right-sized tool.| Model | Size | Runs on | Median latency | Headline result | Catch |
|---|
| Cloudflare Clef | 27B (Qwen3.8 backbone) | Workers AI, Ollama (clef:27b, ~18–20 GB), HF | 209 ms | BANKING77 macro-F1 94.2 vs Jev 79.7; leads the community Jev Decision Index | Loses to Jev on When2Call (72.4 vs 81.0) |
| Cloudflare Clef-flash | ~9B (Qwen3.5 backbone) | Workers AI, HF | 39 ms | BFCL exact 98.8 at $0.038/M input | Falls apart on out-of-scope detection (CLINC150+OOS 66.8) |
| Liquid d1-3B | 3B | llama.cpp, Jetson, HF | 8 ms (RTX 4090), 30 ms (M5 Pro) | 48.6 on Jev Decision Index v0.2.1 — best under 10B per Liquid | AA measured a non-hallucination rate of just 2 — it almost never abstains |
| Liquid d1-omni-600M | 600M | Edge / Jetson | — | Text + image or text + audio input | Experimental; no audio decision benchmarks yet |
| Strands Decider 2B | 2B (Qwen3.5-2B base) | CPU or GPU, pip install strands-decider | ~115 ms (RTX 3090), ~153 ms (M3) | JevBench 0.723, ECE 0.050 — very well calibrated | Can't generate text; weaker on hard multi-step problems |
| Laya | 421M / 322M encoders | Any GPU, pip install laya | 33 ms single, 7 ms batched | 0.766 typed-decisions vs Jev 0.727 | That number needs fine-tuning; zero-shot is ~0.35, and it degrades past ~20 labels |
2. Speed and efficiency: two different routes to "cheaper per answer"
Mercury 2.5 — diffusion decoding at ~1,100 tok/s
Inception's Mercury 2.5 (Sep 8) is a diffusion LLM: rather than emitting one token at a time, it drafts and refines many tokens in parallel. Inception measures ~1,107 tokens/sec on standard NVIDIA GPUs, with a 260K context, tunable reasoning, parallel tool calls, and schema-aligned JSON. List price is $0.20 / $0.75 per million tokens, and a launch promo currently cuts that to $0.04 / $0.15 on Inception and OpenRouter.Inception positions quality alongside GPT-5.6 Luna (Low), Gemini 3.5 Flash-Lite, and Claude Haiku 4.5 — but doesn't publish the scores to back it. AA has not independently benchmarked it yet. The most concrete evidence is a customer claim: Augment Code says context compaction dropped from ~150 s to 27 s at 90% lower cost. That is exactly the workload where Mercury makes sense — high-volume, latency-bound, mid-difficulty transforms like summarization, compaction, extraction, and voice turn-taking. Inception also previewed Mercury Voice and a Mercury Router.Ember-1 — the same Kimi K3, thinking less
Fireworks' Ember-1 (Sep 23) takes the opposite approach: keep a frontier open-weight model (Kimi K3) and post-train it to stop over-thinking. Fireworks reports 35–71% fewer reasoning tokens while holding quality:| Benchmark | Kimi K3 Max | Ember-1 |
|---|
| Terminal Bench 2.1 | 80.9 | 82.0 |
| SWE-bench Verified | 93.2 | 92.2 |
| DeepSWE 1.1 | 66.4 | 75.2 |
| τ²-Bench Airline | 64 | 66 |
3. Open-weight frontier from outside China
Through the summer the open-weight leaderboard was almost entirely Chinese (GLM, Kimi, Qwen, DeepSeek, MiMo). October brought three serious Western contenders. None of them overtakes the Chinese leaders on AA, but each fills a real gap.Reflection Beam — the second-best US open model
Beam (Oct 5) is Reflection AI's first release: a 501B-total / 23B-active sparse MoE with 1M context, Apache 2.0, with weights reported on Hugging Face from Oct 16 in BF16, FP8, and NVFP4 (plus a speculative-decoding drafter). Vendor numbers are strong — SWE-bench Verified 80.9, SWE-Bench Pro 65.5, GPQA Diamond 90.5 — and AA scored it 30 on the Intelligence Index at launch (v4.3, Oct 16) — second among US open-weight models at the time. On the new v5.0 index it sits at 18. It is also notably token-efficient (~26K output tokens per AA task). Reflection itself concedes Kimi K3 is still ahead on raw capability.Mistral Large 4 ("Le Chonk") — Europe's open flagship
Mistral Large 4 (preview Oct 6, weights Oct 29) is a natively multimodal ~1T-total / 52B-active MoE trained in Mistral's European datacenters, covering 160+ languages including every official EU language. At $0.70 / $2.10 it is one of the cheapest large models with an AA score. The preview scored 38 on AA's Intelligence Index v4.3.2 at launch — the top non-Chinese open model at that point. AA has since moved to Index v5.0 (Oct 27), which leans harder on private agentic tests and rescales every model; Large 4's page now shows 26, so only compare it against other v5.0 numbers. Two caveats: it is very verbose (roughly 2.5× the median output tokens on AA's suite), and the weights ship under the Mistral Ecosystem Open-Weight License, a modified MIT that is open-weight but not OSI open source.Aleph Alpha Kolibri-1 — sovereign, bilingual, and very sparse
Kolibri-1 (Oct 3) is a German-built English/German MoE aimed at public administration, industry, and aerospace: 78.1B total but only 3.46B active, Apache 2.0, with a bilingual tokenizer tuned for German compounds and explicit abstention training. Vendor scores are eye-catching for the active size (AIME 2025 96.9, GPQA Diamond 84.3, German GPQA 81.3), but agentic numbers are modest (Terminal-Bench 2.1 27.7) and it still trails Qwen3.6-35B-A3B on AA's hallucination metrics. There is no AA Intelligence Index score yet. Serving needs Aleph Alpha's vLLM plugin and about 78 GB of FP8 weights — one H200/B200 or two H100s.What I'd actually use these for
| If you need… | Try | Why |
|---|
| Fast, cheap routing / classification / guardrails | Clef-flash (hosted) or Liquid d1-3B (on-device) | Millisecond latency, typed outputs, calibrated probabilities |
| A drop-in Jev alternative with vision | Clef | Jev-compatible API, image input, runs locally via Ollama |
| An open decision-model recipe to fine-tune | Strands Decider 2B | Data and training scripts published; best calibration in its class |
| Summarization / compaction at huge volume | Mercury 2.5 | ~1,100 tok/s at promo prices — benchmark it on your own data first |
| Cheaper Kimi K3 coding agents | Ember-1 | Same quality on coding benchmarks with 35–71% fewer reasoning tokens |
| A US-origin open-weight coding model | Reflection Beam | Apache 2.0, strong SWE-bench, token-efficient |
| EU-hosted, EU-language frontier-class model | Mistral Large 4 | 160+ languages, $0.70 / $2.10, open weights |
| German-first, on-prem regulated deployments | Kolibri-1 | Bilingual tokenizer, abstention training, Apache 2.0 |
Themes worth watching
- Calibration is the new headline metric. Decision-model launches lead with Brier scores and expected calibration error, not just accuracy. Expect that to spread to tool-calling and guardrail evals for regular LLMs too.
- Tokens-per-task now matters as much as price-per-token. Ember-1 and Beam compete on doing less work per answer; Mistral Large 4 gets dinged for verbosity. Compare total cost per task, not list price.
- "1M context" usually means "trained at 256K." Kolibri, Beam, and Clef all advertise more context than they were trained or RL'd on. Test long-context recall before you depend on it.
- Sparse MoE is everywhere. Kolibri (~4.4% active), Beam (~4.6%), and Mistral Large 4 (~5%) all activate a small slice of their weights, which is what makes 500B–1T open models practical to serve.
- Independent numbers are still thin — and the yardstick moved. Only Beam and Mistral Large 4 have AA Intelligence Index scores; Mercury 2.5, Ember-1, and Kolibri don't yet. AA also switched to Index v5.0 on Oct 27, so launch-week scores from September and early October aren't comparable with today's leaderboard. Treat every other figure in this post as a vendor claim until it is reproduced.
Sources
