EMO: Pretraining Mixture of Experts for Emergent Modularity
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 7 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Ryan Wang et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2605.06663 |
| Download | |
| Code | https://github.com/allenai/EMO |
Abstract
Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser. We introduce EMO, an MoE designed for modularity-the independent use and composition of expert subsets-without requiring human-defined priors. Our key idea is to encourage tokens from similar domains to rely on similar experts. Since tokens within a document often share a domain, EMO restricts them to select experts from a shared pool, while allowing different documents to use different pools. This simple constraint enables coherent expert groupings to emerge during pretraining using document boundaries alone. We pretrain a 1B-active, 14B-total EMO on 1T tokens. As a full model, it matches standard MoE performance. Crucially, it enables selective expert use: retaining only 25% (12.5%) of experts incurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting. We further find that expert subsets in EMO specialize at semantic levels (e.g., domains such as math or code), in contrast to the low-level syntactic specialization observed in standard MoEs. Altogether, our results demonstrate a path toward modular, memory-efficient deployment of large, sparse models and open new opportunities for composable architectures.
Engineering Breakdown
Plain English
EMO is a new Mixture-of-Experts architecture that solves a practical problem: standard MoEs require running the full model even when you only need specific capabilities (like math or code), defeating the purpose of sparse computation. The paper shows you can train MoEs to naturally cluster—tokens from similar domains automatically route to overlapping expert subsets—without needing human annotation, enabling true modular deployment where you run only relevant experts for a given task.
Key Engineering Insight
The core innovation is removing the need for human-defined domain labels or priors. By designing the training objective to encourage tokens from similar domains to converge on shared experts, the model discovers modularity emergently during pretraining. This means you get composable, domain-specific expert subsets 'for free' without manual partitioning or post-hoc pruning.
Why It Matters for Engineers
This directly addresses memory and inference cost constraints in production. Instead of deploying a 200B parameter model when you only serve code-completion tasks, you can package and run a small expert subset, cutting memory, latency, and hardware costs significantly. It also enables runtime composition—swapping expert combinations for different tasks without retraining, which is valuable for multi-tenant or adaptive systems.
Research Context
Prior MoE work showed that restricting inference to domain-specific experts causes steep performance drops because experts are entangled and task-agnostic. EMO advances this by proving you can align expert specialization with semantic/domain boundaries during pretraining via a clever routing objective. This enables the promised MoE benefit—sparse, modular inference—that was theoretical before, bridging the gap between MoE's sparse potential and practical single-model deployment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::