Inside Ten Open-Weight LLMs: An Architecture Explainer

Ten open-weight LLMs from early 2026 grouped by attention design: full grouped-query attention, sliding-window hybrids, latent and sparse attention, and linear-attention hybrids, with active parameters as a share of total
In the space of about three weeks last January and February, ten notable open-weight language models shipped: from a 3B model that runs on a laptop to two 1-trillion-parameter giants. Sebastian Raschka put all ten side by side in one figure, and it is one of the best snapshots I've seen of how modern LLMs are actually built. This post is a plain-language companion to that figure. It explains each building block once, then walks through what each model does with them, using numbers taken from the published model configs. The surprising takeaway is how few ideas there are. Almost every design is the same transformer with different choices on two dials: how sparse the feed-forward layers are, and how expensive attention is allowed to be.
Raschka's original write-up, A Dream of Spring for Open-Weight LLMs, and his LLM Architecture Gallery are worth reading alongside this. Where his figure and the configs disagree, I've gone with the config.json published with each model on Hugging Face.

The ten models at a glance

ModelLabReleasedTotal / active paramsLayersAttention designExperts (routed + shared)Context
Kimi K2.5Moonshot AIJan 271.03T / 32B61MLA (latent)8 of 384 + 1256K
Ling 2.5Ant Group (inclusionAI)Feb 161T / 63B80Lightning linear + MLA, 7:18 of 256 + 1256K (1M with YaRN)
GLM-5Z.aiFeb 11744B / 40B78MLA + DeepSeek Sparse Attention8 of 256 + 1~200K
Trinity LargeArcee AIJan 27398B / 13B60GQA, sliding window 3:14 of 256 + 1256K
Qwen3.5AlibabaFeb 16397B / 17B60Gated DeltaNet + gated attention, 3:110 of 512 + 1256K
MiniMax-M2.5MiniMaxFeb 12230B / 10B62Full GQA8 of 256, no shared192K
Step 3.5 FlashStepFunFeb 1199B / 11B45GQA, sliding window 3:18 of 288 + 1256K
Qwen3-Coder-NextAlibabaFeb 380B / 3B48Gated DeltaNet + gated attention, 3:110 of 512 + 1256K
Tiny AyaCohere LabsFeb 173.35B dense36GQA, sliding window + global(dense)8K
Nanbeige 4.1NanbeigeFeb 103B dense32Full GQA(dense)256K
All ten are decoder-only transformers: text goes in as tokens, passes through a stack of identical-looking blocks, and comes out as a probability over the next token. Each block has two halves, an attention layer (tokens look at each other) and a feed-forward layer (each token is processed on its own). Nearly every interesting difference between these models is in one of those two halves.

Part 1: The feed-forward half — mixture-of-experts

Dense vs. mixture-of-experts

In a classic "dense" model, every token goes through the same big feed-forward network. Tiny Aya and Nanbeige 4.1 still work this way. The other eight replace that one network with a mixture-of-experts (MoE) layer: hundreds of small feed-forward networks ("experts"), plus a tiny router that picks a handful of them for each token.That gives you two parameter counts. Total parameters decide how much knowledge the model can store and how much memory you need to hold it. Active parameters decide how much compute each token costs. MoE lets those two numbers drift far apart:
ModelActive share of total
Qwen3-Coder-Next3B of 80B ≈ 3.8%
Kimi K2.532B of 1.03T ≈ 3.1%
Trinity Large13B of 398B ≈ 3.3%
Qwen3.517B of 397B ≈ 4.3%
MiniMax-M2.510B of 230B ≈ 4.3%
GLM-540B of 744B ≈ 5.4%
Step 3.5 Flash11B of 199B ≈ 5.5%
Ling 2.563B of 1T ≈ 6.3%
For comparison, Mixtral 8x7B, which popularized open MoE in late 2023, used about 28% of its weights per token. In early 2026 most frontier-scale open models touched only 3–6%. That is why a 1T-parameter model like Kimi K2.5 can serve tokens at roughly the cost of a 32B dense model. The catch is that you still have to fit all 1T parameters in memory somewhere.

Many small experts

The trend is toward more, smaller experts. Qwen3.5 and Qwen3-Coder-Next have 512 experts each and route every token to 10 of them. Kimi K2.5 has 384. Each individual expert is small: Qwen3-Coder-Next's experts have a hidden size of just 512. Finer-grained experts let the router assemble a more specific combination for each token, a "team" of specialists instead of a few generalists.

The shared expert

Seven of the eight MoE models also run one shared expert that every token goes through regardless of routing. The idea, popularized by DeepSeek, is that some knowledge (basic grammar, common patterns) is useful for every token. Giving it a permanent home frees the routed experts to specialize. MiniMax-M2.5 is the exception: it has 256 experts, picks 8, and has no shared expert at all.

Dense layers first

Several models keep their first few blocks dense before switching to MoE: Trinity (6 blocks), Ling 2.5 (4), GLM-5 (3), Step 3.5 Flash (3) and Kimi K2.5 (1). Early layers handle low-level features that every token needs. Routing there tends to be unstable and saves little, so a plain feed-forward layer is simpler and trains more reliably.

Part 2: The attention half — where the real variety is

Attention is how each token gathers information from earlier tokens. Standard attention compares every token with every other token, so its cost grows with the square of the context length. It also needs a KV cache, a stored copy of the "keys" and "values" for every past token. At 256K tokens, that cache, not the model weights, is often what fills your GPU. Every attention variant below is a different answer to the question "how much of that cost can we cut without hurting quality?"

Grouped-query attention (GQA): the baseline

Attention has multiple "heads" that each look for different patterns. GQA lets several query heads share one set of keys and values, which shrinks the KV cache. MiniMax-M2.5 uses 48 query heads but only 8 key/value heads, a 6× smaller cache than giving every head its own. Nanbeige 4.1 uses 20 query heads and 4 key/value heads. In the figure, GQA is the "default" choice, and MiniMax-M2.5 and Nanbeige 4.1 use it in every layer, with no other tricks. Both are strong for their size, which is a useful reminder that plain designs still compete.

Sliding-window attention: look only at recent tokens in most layers

Sliding-window attention (SWA) restricts a layer to the last N tokens rather than the whole context. The cost no longer grows with context length for that layer. Models interleave these "local" layers with occasional "global" layers that see everything, so information can still flow across long documents.
  • Trinity Large: 3 local layers for every global one (45 local, 15 global), with a 4,096-token window. Its global layers use NoPE (no positional encoding), relying on the local layers to supply position information. This tends to help the model generalize to longer contexts than it was trained on.
  • Step 3.5 Flash: also 3:1, but with a very small 512-token window. To compensate, its local layers get more attention heads (96) than its global layers (64).
  • Tiny Aya: Cohere's Command-A-style mix of 4,096-token sliding-window layers and global layers.

Multi-head latent attention (MLA): compress the cache

MLA, introduced by DeepSeek, takes a different approach. Instead of storing full keys and values, it stores a small compressed ("latent") vector per token and reconstructs keys and values from it when needed. The KV cache shrinks dramatically while every layer can still see the full context. Kimi K2.5, GLM-5 and Ling 2.5 all use MLA. Kimi K2.5 is essentially DeepSeek V3's architecture scaled up to 1T parameters.

Sparse attention: only attend to the tokens that matter

GLM-5 adds DeepSeek Sparse Attention (DSA) on top of MLA. A lightweight "indexer" scans the context and picks the 2,048 most relevant tokens, and full attention runs only over those. At long context lengths, that turns quadratic cost into something close to linear while keeping full-attention quality for the tokens that count.

Linear attention: replace most attention layers with a running memory

The most radical option drops standard attention in most layers. Linear attention keeps a fixed-size running state that is updated as each token arrives, much like a recurrent network. Cost per token stays constant no matter how long the context is, and there is no growing KV cache for those layers. The trade-off is that a fixed-size state can't recall everything precisely, so models keep a few full-attention layers to handle exact lookups.
  • Qwen3.5 and Qwen3-Coder-Next use Gated DeltaNet, a linear-attention layer with a learned "forget gate", in 3 of every 4 layers. The fourth is a gated full-attention layer. Qwen3.5's 60 layers split into 45 DeltaNet and 15 attention layers.
  • Ling 2.5 goes further, with a 7:1 ratio of Lightning attention (another linear-attention variant) to MLA. Notably, Ant Group didn't train it that way from scratch: they converted Ling 2.0's standard attention into the hybrid through continued training, and pushed the usable context to 1M tokens.

Spectrum of attention cost

Arranged from most to least expensive at long context:
  1. Full GQA in every layer: MiniMax-M2.5, Nanbeige 4.1
  2. Full MLA in every layer: Kimi K2.5
  3. Sliding window + periodic global: Trinity Large, Step 3.5 Flash, Tiny Aya
  4. MLA + sparse token selection: GLM-5
  5. Linear attention with periodic full attention: Qwen3.5, Qwen3-Coder-Next (3:1), Ling 2.5 (7:1)

Part 3: The small tweaks that add up

Gated attention

Several models add a learned gate on the attention output, a per-head switch that can dim or silence that head's contribution. Qwen3.5, Qwen3-Coder-Next, Step 3.5 Flash and Trinity Large all use some form of it. It improves training stability and helps avoid "attention sinks", where heads dump attention on the first token when they have nothing useful to attend to.

QK-norm and sandwich norms

QK-norm normalizes queries and keys before they are compared, which prevents the attention scores from blowing up during training. MiniMax-M2.5, Step 3.5 Flash and Trinity Large use it. Trinity also uses a "sandwich" of normalization layers before and after each sub-layer. Both are cheap insurance for training very large models without loss spikes.

Multi-token prediction (MTP)

Normally a model predicts one token at a time. MTP adds small extra heads that also guess the next few tokens. During training this gives a richer learning signal. At inference it can act as built-in speculative decoding: draft several tokens, verify them in one pass, and keep the ones that are right. GLM-5 and Qwen3.5 have an MTP layer, and MiniMax-M2.5's config ships three. Step 3.5 Flash stands out for using MTP-3 at inference to speed up generation, not just as a training aid.

Parallel blocks

Tiny Aya runs attention and feed-forward in parallel within each block rather than one after the other, then adds both results back together. It is a GPT-J / Cohere Command idea that improves hardware utilization at small scale.

Part 4: Model by model

Kimi K2.5 (Moonshot AI). DeepSeek V3's recipe at 1T scale: MLA in every layer, 384 experts with 8 routed plus 1 shared, and one dense block at the bottom. The text backbone is unchanged from Kimi K2. The headline change is native multimodality: a vision encoder fused in early and trained on roughly 15T mixed vision and text tokens.Ling 2.5 (Ant Group / inclusionAI). The largest active count of the group at 63B, up from 51B in Ling 2.0, across 80 layers with a hidden size of 8,192. Its 7:1 Lightning/MLA hybrid is the most aggressive bet on linear attention in the set, aimed at million-token contexts and higher throughput.GLM-5 (Z.ai). Z.ai switched to DeepSeek's playbook: MLA plus DeepSeek Sparse Attention, 256 experts, and a multi-token prediction layer. It is shallower than GLM-4.7 (78 vs. 92 layers) but wider, and sits at about 40B active out of 744B.Trinity Large (Arcee AI). A US-built entry with a Gemma-3-style local/global attention pattern, NoPE global layers, gated attention, QK-norm and sandwich norms, on a very sparse MoE: 4 of 256 experts plus a shared one, for about 13B active out of 398B.Qwen3.5 (Alibaba). The Qwen3-Next hybrid recipe (Gated DeltaNet plus gated attention at 3:1) scaled to a 397B flagship with 512 experts, a 248K-token vocabulary that suits its multilingual focus, and native multimodality.MiniMax-M2.5 (MiniMax). Deliberately conventional: full GQA everywhere, QK-norm, partial RoPE, 8 of 256 experts with no shared expert. It competes on cost, at about 10B active, rather than architectural novelty.Step 3.5 Flash (StepFun). Built for speed: a 512-token sliding window in three of every four layers, extra heads in the local layers, a head-wise output gate, and MTP-3 at inference. 288 experts, 8 routed plus 1 shared, at about 11B active.Qwen3-Coder-Next (Alibaba). The same hybrid architecture as Qwen3.5 in miniature: 80B total but only 3B active, with 512 tiny experts. It shows how far a linear-attention coder can go on a fraction of the compute.Tiny Aya (Cohere Labs). A 3.35B dense multilingual model covering about 70 languages, with Command-A-style parallel blocks and sliding-window layers. It ships as several region-tuned variants (Global, Earth, Fire, Water). Note the CC-BY-NC license: it is not for commercial use.Nanbeige 4.1 (Nanbeige). A Llama-3.2-style 3B model with full GQA (20 query and 4 key/value heads), untied embeddings and a 256K context. As Raschka points out, almost all of its gains come from training data, supervised fine-tuning and reinforcement learning, not architecture.

What these ten tell us

  1. The transformer is the constant. None of these is a fundamentally new architecture. They are all GPT-lineage decoders with different choices for the feed-forward and attention halves. Most of the performance difference between them comes from data and post-training, not from the diagram.
  2. Sparsity is now the default at scale. Every model above about 50B parameters is MoE, typically with 3–6% of weights active, many small experts and one shared expert.
  3. Attention is where labs differentiate. Cheap-but-complete (GQA, MLA) and cheap-but-approximate (sliding windows, sparse selection, linear attention) all coexist. The hybrids, which mix a cheap layer type with a few full-attention layers, are clearly gaining ground, mainly because long-context agent workloads make the KV cache the bottleneck.
  4. Serving cost shapes the design. Partial RoPE, MTP at inference, tiny sliding windows and huge expert counts are all optimizations for tokens per dollar, not benchmark scores.
  5. Plain still works. MiniMax-M2.5 and Nanbeige 4.1 used nothing exotic and stayed competitive. A clever architecture can't replace a good training recipe.

Nine months later

Most of these labs have shipped successors since. In my LLM Landscape leaderboard, Kimi K3.1 Pro is now the strongest model you can download, and the open-weight tier also includes GLM-5.3, Qwen3.8, MiniMax-M3.1-Flash and Ling 3.1 Flash. The vocabulary in this post (active parameters, shared experts, latent, sliding-window, sparse and linear attention, MTP) is still how you read their model cards. When a new open-weight model ships, the two questions worth asking are the same as in February: how much of the model does each token touch, and how much does attention cost at long context. The answers tell you most of what you need to know about how expensive the model will be to run.

Sources: Sebastian Raschka, “A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan–Feb 2026” and the LLM Architecture Gallery; Hugging Face model cards and config.json files for Kimi K2.5, Ling 2.5 1T, GLM-5, Trinity Large, Qwen3.5-397B-A17B, MiniMax-M2.5, Step 3.5 Flash, Qwen3-Coder-Next, Tiny Aya and Nanbeige 4.1; Arcee's Trinity technical report and Cohere's Tiny Aya paper. Parameter counts are rounded. Active-parameter shares are computed from the published totals.