Mixture of Experts (MoE)

Why This Matters

When a model card says "671B parameters, 37B active," that is MoE, and the gap between those numbers explains modern AI economics. MoE is why open models like DeepSeek-V3 and Mixtral punch far above their compute cost, why DeepSeek could report a ~$5.6M pre-training compute bill for a frontier-class model, and why serving infrastructure needs enormous memory even when per-token compute is modest. If you can't read the total/active split, you will mis-size hardware by an order of magnitude: budgeting GPUs for a "13B-fast" Mixtral and discovering it needs 47B parameters' worth of memory, or comparing a 671B MoE to a 405B dense model as if the bigger number meant more compute per token (it means less). Reading model specs correctly requires understanding this architecture.

Everyday Analogy

A hospital does not send every patient to one all-knowing doctor. A triage desk routes each patient to two or three relevant specialists among dozens on staff. The hospital's total expertise is huge; the effort spent per patient is small. In an MoE model, the router is the triage desk, experts are the specialists, and each token is a patient. The analogy even captures the costs: the hospital must pay to keep every specialist on site whether or not today's patients need them (memory), and if triage funnels everyone to the same two doctors, those doctors drown while the rest idle (routing collapse).

How MoE Works

In a standard ("dense") transformer, every token flows through every parameter. Recall from AI-012 that a transformer block is attention followed by a feed-forward network (FFN), and the FFN typically holds about two-thirds of the block's parameters. MoE targets exactly that layer:

1. Each MoE layer holds many experts (8 to 256 small feed-forward networks, each structurally identical to a normal FFN).
2. A tiny router network (often a single linear layer) scores each incoming token against every expert and picks the top-k (usually k = 1, 2, or 8 depending on design).
3. Only those experts run for that token; the rest stay idle for it.
4. Outputs are combined, weighted by the router's normalized scores.

Attention layers usually stay dense; sparsity applies to the FFN, which is where the parameter bulk lives. A model can therefore have massive total parameters (all experts) while each token touches only a small active subset:

DeepSeek-V3 stores nearly 10× the parameters of a dense 70B model while spending roughly half the per-token compute. Note the design trend from Mixtral to DeepSeek: fine-grained experts, meaning many more, much smaller experts, plus a shared expert that every token passes through to carry common knowledge, freeing the routed experts to specialize harder.

The Active-vs-Total Math

Two numbers, two different resources. Work through DeepSeek-V3 concretely:

• Compute per token ∝ active parameters. A forward pass costs roughly 2 FLOPs per active parameter per token. DeepSeek-V3: 2 × 37B ≈ 74 GFLOPs/token. A dense 405B model (Llama 3.1 405B): 2 × 405B ≈ 810 GFLOPs/token, about 11× more compute for every single token, forever, at any scale of serving.
• Memory ∝ total parameters. At fp16 (2 bytes/parameter; see AI-065), DeepSeek-V3 needs 671B × 2 B ≈ 1.34 TB just for weights, which means a multi-node deployment (e.g., 16+ H100 80GB GPUs before you count KV cache). Mixtral 8x7B needs 47B × 2 B ≈ 94 GB, so two 80GB GPUs, or one with 4-bit quantization (≈ 24 GB).
• Training compute ∝ active parameters × tokens. This is the economic punchline. DeepSeek-V3 pre-trained on 14.8T tokens; because only 37B parameters activate per token, the reported cost was ~2.79M H800 GPU-hours ≈ $5.6M at their rental rate (their figure for the final pre-training run, excluding research, ablations, staff, and infrastructure). A dense model with 671B always-on parameters on the same data would have cost roughly 18× more compute.

The one-line summary: total parameters buy knowledge capacity; active parameters set the per-token bill; MoE lets you buy the first without paying the second.

Inside the Router

The router is disarmingly small. For a hidden size of 4,096 and 64 experts, it is a single 4,096 × 64 matrix, about 262K parameters deciding how to spend billions. Trace one token through a top-2 layer:

1. The token's hidden vector h is multiplied by the router matrix, producing 64 logits, one score per expert.
2. Top-2 selection keeps the two highest, say expert 17 (logit 2.1) and expert 42 (logit 1.3), and drops the rest.
3. A softmax over just those two gives weights ≈ 0.69 and 0.31.
4. The token runs through experts 17 and 42 only; the layer output is 0.69 × E17(h) + 0.31 × E42(h).

Because top-k selection is a hard, non-differentiable choice, gradients flow through the softmax weights of the chosen experts. The router learns "given whom I picked, how should I have weighted them," and over many batches this is enough to shape whom it picks. It works, but it is the fragile part of the architecture: early mistakes compound, which is exactly why routing collapse (below) needs explicit countermeasures.

Design choices you will see on model cards:

• k (experts per token). Switch Transformer showed even k = 1 works; Mixtral uses k = 2; DeepSeek-V3 uses k = 8 out of 256. Higher k means more compute per token but smoother training and blending.
• Expert granularity. 8 big experts (Mixtral) vs 256 small ones (DeepSeek-V3) at similar active compute: finer granularity gives the router combinatorially more mixtures (choose 8 of 256 rather than 2 of 8) at the cost of harder balancing and more communication.
• Shared experts. DeepSeek routes every token through one always-on expert plus its top-8. Common knowledge (grammar, basic facts) lives in the shared path, so routed experts don't all redundantly learn it.

A Short History (Why It Took So Long)

The idea is old (Jacobs and Jordan proposed mixtures of experts in 1991), but three modern milestones made it practical at LLM scale. Shazeer et al. (2017) introduced the sparsely-gated MoE layer with noisy top-k routing and load-balancing losses, scaling LSTMs to 137B parameters. Google's GShard (2020) and Switch Transformer (2021) moved MoE into transformers and showed a 1.6T-parameter sparse model pre-training ~7× faster than a dense T5 baseline at equal quality per FLOP. Then Mixtral (2023) and DeepSeek-V2/V3 (2024) proved the recipe in open weights, at which point the total/active split became standard model-card vocabulary. The lag between 1991 and 2023 was mostly systems, not ideas: sparse routing only pays off once models are big enough that FFN compute dominates and clusters are large enough to hold all experts in fast memory.

What Experts Actually Learn

Experts do not become tidy human categories ("the math expert," "the French expert"). Routing is learned end-to-end, and specialization emerges along statistical lines that often surprise people. The Mixtral paper's own analysis found little evidence of topic-level specialization. Instead, experts showed preferences for token types and positions: one expert catching whitespace and indentation in code, another favoring numbers, with consecutive tokens often routed to the same expert. Fine-grained designs like DeepSeek's push further toward micro-specializations. The router and experts co-evolve during training: experts get good at what the router sends them, and the router learns to send tokens where they are processed best. This is why you cannot delete "the experts you don't need" to shrink a model: the specializations don't align with your use case's boundaries.

The Trade-off Table

The details behind the risky rows:

• Routing collapse. Early in training the router may find a few "good enough" experts and send everything there; those experts improve, attracting even more traffic, a rich-get-richer loop that wastes most of the model. The standard fix is an auxiliary load-balancing loss penalizing uneven expert utilization; DeepSeek-V3 refined this with an auxiliary-loss-free bias-adjustment scheme to avoid the quality tax the auxiliary loss itself imposes.
• Serving complexity. With 256 experts spread across GPUs, each batch of tokens must be scattered to the devices holding its chosen experts and gathered back, an all-to-all communication pattern that becomes the bottleneck if expert placement and batching are naive. This is engineering you inherit if you self-host a big MoE, and it's invisible if you use an API.
• Capacity overflow. Hardware wants predictable shapes, so implementations often cap tokens per expert per batch (a capacity factor); overflow tokens get dropped from that expert or rerouted, a quality-vs-efficiency knob you tune at training time.

Two mitigations worth knowing by name. Upcycling initializes an MoE from a trained dense model: copy the dense FFN into every expert, add a router, and continue training, so you skip the most unstable early phase; several open MoE models (e.g., Qwen's early MoE releases) started this way rather than from scratch. And expert parallelism is the serving pattern that makes big MoE tractable: place different experts on different GPUs so each device holds a slice of the total parameters, then batch aggressively so every expert sees enough tokens per step to stay efficient. Both are reasons MoE favors organizations with fleets: the tricks amortize at scale and mostly don't help a hobbyist with one card.

Worked Example: Sizing Mixtral 8x7B for Deployment

You are asked whether Mixtral 8x7B fits your single 24 GB RTX 4090, and what speed to expect. Trace it:

1. Decode the name. "8x7B" = 8 experts per layer, each layer's expert about 7B-scale. Total is 47B, not 8 × 7 = 56B, because attention layers and embeddings are shared, not multiplied.
2. Memory at fp16: 47B × 2 bytes = 94 GB. Does not fit, not even close.
3. Memory at 4-bit quantization (see AI-065): 47B × 0.5 bytes ≈ 23.5 GB plus KV cache and overhead. Borderline on 24 GB, feasible with a small context, comfortable on a 48 GB card.
4. Speed: top-2 routing activates ~13B parameters per token, so decode throughput resembles a 13B dense model, not a 47B one. Memory-bandwidth math (see AI-065's worked example) applies to the active bytes read per token.
5. The comparison that matters: against a dense 13B model, Mixtral costs ~3.6× the memory for the same speed but substantially higher quality: Mistral's release benchmarks showed it matching or beating Llama 2 70B (a model with 5.4× its active parameters) on most tasks while generating ~6× faster.

That is the MoE bargain in one deployment decision: pay memory, save compute, gain quality.

Decision Framework: Dense or MoE?

• Consuming via API? The architecture is mostly the provider's problem. Read total/active only to sanity-check pricing and latency claims: active params predict speed and cost, total predicts capability ceiling.
• Self-hosting on one modest GPU? Prefer dense (or small MoE like Mixtral quantized). MoE's memory-for-compute trade is exactly backwards when memory is your scarce resource.
• Serving at high throughput on a GPU cluster? MoE wins: memory is amortized across the fleet, and per-token compute savings multiply across millions of requests. This is why every frontier lab converged on it.
• Training your own foundation model? MoE if you have the data and engineering depth, since quality per training FLOP is decisively better (Switch Transformer's ~7× speedup; DeepSeek-V3's $5.6M run). Dense if your team is small: fewer failure modes, simpler tooling, easier fine-tuning.
• Fine-tuning an open model? Dense checkpoints are the well-trodden path; MoE fine-tuning works but budget extra time for stability issues.

Real-World Showcase

• Mixtral 8x7B (December 2023) brought MoE to open source: 47B total, ~13B active, matching or beating Llama 2 70B on most benchmarks with ~6× faster inference. This was the moment MoE went mainstream outside the big labs.
• DeepSeek-V3 (December 2024) trained a 671B-total, 37B-active MoE on 14.8T tokens for a reported ~$5.6M in pre-training compute, an order of magnitude below comparable dense frontier training, largely thanks to sparse activation (and the base for the R1 reasoning model; see AI-063).
• GPT-4 (2023) is widely reported (never officially confirmed) to be a large MoE. The architecture quietly powered frontier models for years before the open ecosystem caught up; Google's Switch Transformer had already demonstrated trillion-plus-parameter sparse models in 2021.
• Llama 4 (2025) marked Meta's switch from dense (Llama 1–3) to MoE: Maverick pairs ~400B total with just 17B active. When the last major dense holdout converts, the architectural argument is effectively settled at frontier scale.

Common Misconceptions

• "A 671B MoE is 'bigger' than a 405B dense model, so it must cost more to run per token." Backwards: DeepSeek-V3's 37B active parameters mean ~11× less compute per token than dense 405B. Total parameters are a memory number, not a speed number.
• "8 experts means 8 domain specialists: math, code, French…" Specialization is statistical and emergent; Mixtral's analysis found token-type and syntax patterns, not human topics. You cannot prune "irrelevant" experts for your domain.
• "Low active parameters means it runs on a small GPU." Every expert must be resident in memory. Mixtral's 13B-active still demands 47B parameters' worth of VRAM (94 GB fp16).
• "Sparse means lower quality, since it's skipping most of the network." At equal training compute, MoE consistently matches or beats dense; skipping is learned, routing each token to the parameters most useful for it.
• "The router is a fixed rulebook." The router is trained jointly with the experts and can misbehave: routing collapse is a real failure mode that load-balancing objectives exist to prevent.

Try It Yourself

1. Read a model card ("Mixtral 8x7B: 47B total, 13B active") and answer two questions. How much GPU memory does it need: sized by 47B or 13B? (47B, since every expert loads.) How fast does it generate: like a 47B or a 13B model? (Like ~13B, since only active parameters compute.) That two-number reading habit is the practical skill this lesson builds.
2. Redo the worked example for DeepSeek-V3: compute fp16 weight memory (671B × 2 B) and per-token FLOPs (2 × 37B), then compare both against Llama 3.1 405B dense. Which resource does each model stress?
3. Find the total/active numbers for two recent open MoE releases (e.g., Llama 4, Qwen3-MoE) and compute their sparsity ratio (active ÷ total). Notice how the trend has moved from Mixtral's ~28% toward 5% and below.

Mini Project

Build a toy MoE layer in ~80 lines of NumPy or PyTorch. Create 8 small two-layer MLPs (the experts) and a linear router. For each input vector: softmax the router scores, pick top-2, run only those experts, and combine outputs weighted by the renormalized scores. Train it on a synthetic task with visibly different clusters (e.g., regression where inputs from cluster A follow one function and cluster B another). Then instrument it: log tokens-per-expert per batch. First run it with no balancing and watch utilization skew toward a few experts; then add a load-balancing penalty (variance of expert usage) and watch it even out. Report: final loss with and without balancing, and the expert-utilization histogram for each. You will have reproduced, in miniature, both the mechanism and its most famous failure mode.

Key Takeaways

• MoE = many small experts + a learned router; only the top-k experts run per token, and sparsity lives in the feed-forward layers where most parameters sit.
• Total parameters set memory footprint and knowledge capacity; active parameters set per-token compute, speed, and training cost. DeepSeek-V3's 671B/37B split is the canonical example.
• The economics favor MoE at frontier scale (~$5.6M reported pre-training compute for V3; Switch Transformer's ~7× training speedup), which is why most frontier models use it.
• The costs move, they don't vanish: full-model memory, load-balancing to prevent routing collapse, and all-to-all serving complexity replace raw compute as the bottleneck.
• Expert specialization is emergent and statistical (token types, syntax), not thematic, so you cannot carve out a domain-specific subset.
• Deployment rule of thumb: single small GPU → dense; high-throughput cluster or frontier training → MoE.