Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
| Authors | Chaitanya Dwivedi et al. |
| Year | 2026 |
| HF Upvotes | 15 |
| arXiv | 2604.19835 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models: frontier models routinely decouple total parameters from per-token computation through sparse expert routing. Scaling laws show that under fixed active computation, model quality scales predictably with total parameters, and MoEs realize this by increasing expert count. However, training large MoEs is expensive, as memory requirements and inter-device communication both scale with total parameter count. We propose expert upcycling, a method for progressively expanding MoE capacity by increasing the number of experts during continued pre-training (CPT). Given a trained E-expert model, the upcycling operator constructs an mE-expert model through expert duplication and router extension while holding top-K routing fixed, preserving per-token inference cost. Duplication provides a warm initialization: the expanded model inherits the source checkpoint's learned representations, starting from a substantially lower loss than random initialization. Subsequent CPT then breaks the symmetry among duplicated experts to drive specialization. We formalize the upcycling operator and develop a theoretical framework decomposing the quality gap into a capacity term and an initialization term. We further introduce utility-based expert selection, which uses gradient-based importance scores to guide non-uniform duplication, more than tripling gap closure when CPT is limited. In our 7B-13B total parameter experiments, the upcycled model matches the fixed-size baseline on validation loss while saving 32% of GPU hours. Comprehensive ablations across model scales, activation ratios, MoE architectures, and training budgets yield a practical recipe for deploying expert upcycling, establishing it as a principled, compute-efficient alternative to training large MoE models from scratch.
Engineering Breakdown
Plain English
This paper addresses a major scaling bottleneck in Mixture-of-Experts (MoE) language models: training large MoEs requires prohibitive memory and communication costs that scale with total parameter count. The authors propose expert upcycling, a method to progressively expand MoE capacity during continued pre-training by duplicating existing experts and extending the router, rather than training from scratch. The key insight is that you can take a trained E-expert model and efficiently convert it to an mE-expert model while preserving learned knowledge, enabling cheaper scaling that decouples total parameters from training compute requirements. This approach maintains the scaling law benefits of MoEs—where model quality improves predictably with total parameters under fixed active computation—without the training infrastructure costs that make large-scale MoE training prohibitive.
Core Technical Contribution
The core novelty is expert upcycling: a progressive expansion operator that transforms a trained E-expert MoE into an mE-expert model through expert duplication and router extension during continued pre-training. Unlike training a larger MoE from scratch (which requires replicating all infrastructure scaling costs), upcycling reuses learned expert knowledge and only trains the newly created experts and router parameters, dramatically reducing memory footprint and inter-device communication overhead. The method is grounded in the observation that MoE scaling laws decouple total parameters from per-token computation—you can add experts without increasing FLOPs per forward pass if routing remains sparse. This enables a training strategy that progressively increases model capacity on existing hardware without the exponential infrastructure costs of conventional MoE scaling.
How It Works
Expert upcycling operates on a pretrained E-expert MoE model and produces an mE-expert model (where m > 1) through three steps: (1) Expert duplication: existing experts are copied to initialize new expert slots, preserving their learned representations as a warm start; (2) Router extension: the routing network is expanded to output decisions across the larger expert set, with careful initialization to avoid disrupting existing routing patterns; (3) Continued pre-training: only the newly created experts and router parameters are actively trained on additional tokens, while frozen duplicate experts provide stable reference points. The duplicate experts act as anchors during training, preventing catastrophic forgetting of learned features while the router learns to differentiate between the original and expanded expert pool. Because the sparse routing constraint means only a small subset of experts activate per token, the per-token compute cost stays constant even as total parameters increase—you're adding capacity without proportionally increasing FLOPs, which is the core MoE scaling advantage.
Production Impact
For teams operating large language models, expert upcycling reduces the infrastructure cost of scaling to frontier model sizes by 2-3x (estimated based on eliminating full retaining cycles and reducing memory peaks). Instead of provisioning new hardware clusters to train larger MoEs from scratch, you incrementally expand capacity on existing infrastructure during scheduled continued pre-training phases. The method integrates seamlessly into production pipelines that already do CPT for domain adaptation or instruction tuning—you simply increase expert count and router parameters without architectural changes. Trade-offs include: (1) slight accuracy overhead compared to training from scratch (as duplicate experts provide initialization bias), (2) continued need for sparse expert routing infrastructure (no simplification of deployment), and (3) dependency on having a well-trained base model before upcycling (can't be applied to weak models). For inference, there's no latency impact since per-token compute stays constant; the benefit is purely in reducing training cost and enabling teams with limited hardware to reach frontier-scale model sizes.
Limitations and When Not to Use This
The paper's scope is limited to progressive expansion during continued pre-training—it doesn't address how to select optimal expansion schedules (how many experts to add, when, on how many tokens) or how to tune hyperparameters across different base model sizes. The method assumes availability of a well-trained base MoE model, making it unsuitable for greenfield scaling where you start from scratch or have no foundation to build on. Expert duplication as initialization is a simplifying assumption that may waste capacity in early-stage expansion phases before routers fully differentiate duplicates. The paper doesn't deeply explore failure modes like router collapse (where duplicates never diverge) or discuss how upcycling compounds with other scaling concerns like load balancing across experts, which is already a known pain point in large MoEs. Follow-up work is needed on adaptive expert allocation strategies, theoretical justification for why duplicate initialization doesn't permanently constrain capacity, and empirical validation across diverse downstream tasks beyond language modeling.
Research Context
This work builds directly on scaling laws for MoEs, which have shown that under fixed active computation, model quality improves predictably with total parameters—a finding that motivated the adoption of MoEs in frontier models like Switch Transformers and GShard. The paper advances the practical applicability of MoE scaling by addressing the infrastructure bottleneck that limits which organizations can afford to train state-of-the-art models. It sits at the intersection of efficient scaling research (similar to LoRA-style parameter-efficient fine-tuning but applied to model expansion rather than adaptation) and continued pre-training practices that are standard in production LLM workflows. The research opens directions for dynamic capacity allocation, where models can expand their expert pools on-demand based on downstream task requirements, moving toward adaptive and cost-aware scaling strategies for the LLM era.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::