REAM: Merging Improves Pruning of Experts in LLMs
| Authors | Saurav Jha et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.04356 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
Engineering Breakdown
Plain English
This paper addresses the memory challenges of deploying massive Mixture-of-Experts (MoE) language models by proposing REAM (Router-weighted Expert Activation Merging), which groups and merges expert weights instead of pruning them outright. The authors built on prior work called REAP that removes underutilized experts, but discovered that merging experts preserves model performance better than deletion across hundreds of billions of parameters. The key finding is a trade-off: REAM achieves better performance on some benchmarks (multiple-choice QA) while potentially trading off others (generative tasks), offering engineers a tunable compression option for deployed MoE systems.
Core Technical Contribution
REAM's core innovation is replacing expert pruning (deletion) with expert merging (weighted combination), guided by router activation patterns learned during training. Unlike REAP which permanently removes low-activation experts, REAM groups similar experts and combines their weights using router-based importance weighting, creating a denser model that retains more of the original capacity. This is fundamentally different because it avoids the information loss inherent in hard pruning—instead of throwing away parameters, you compress them into a smaller expert set. The router weights themselves become the guide for how to merge, making the process directly aligned with the model's learned routing decisions.
How It Works
REAM operates in three stages: First, during or after training, it analyzes router activation patterns to identify which experts have overlapping or redundant functionality. Second, it groups experts using these activation patterns as a similarity metric, determining which experts should be merged together. Third, for each group, it creates a merged expert by computing a weighted combination of the original expert weights, where the weights are derived from how frequently the router directed tokens to each expert. The merged expert replaces the group in the final model, reducing the total number of experts while maintaining the same dimensional capacity per expert. The output is a smaller MoE model with fewer but richer experts, which can be deployed with lower memory footprint while retaining more capability than pure pruning approaches.
Production Impact
For engineers deploying MoE models, REAM offers a middle ground between full-size models (expensive in memory and compute) and hard-pruned models (which lose capability). Instead of keeping hundreds of experts and paying per-token routing costs, you'd compress to perhaps 50-60% of the original expert count, reducing GPU memory by roughly 35-45% depending on merge ratios, while retaining 90%+ of original performance on many tasks. The trade-off is that generative tasks may see larger performance drops than multiple-choice tasks, requiring benchmark evaluation specific to your workload before deployment. Implementation-wise, REAM requires capturing router statistics during a validation phase and then performing weight merging offline—straightforward engineering with no changes to inference serving code once the merged model is saved. The main production concern is the clear benchmark fragmentation: you must test both MC and generative capabilities in your use case to know if the compression is worth the trade-off.
Limitations and When Not to Use This
The paper reveals a fundamental tension REAM doesn't fully resolve: merging trades performance differently across task types, with no principled way to predict which tasks will suffer in your specific deployment. The approach assumes router patterns from training remain representative of deployment patterns, which may not hold if your inference distribution shifts significantly from training. REAM also requires a validation dataset to compute router activation statistics—the paper doesn't detail sensitivity to this choice or provide guidance for small-data regimes. Additionally, the merging decision itself is somewhat heuristic; the paper doesn't provide theoretical justification for router-weighted merging over other similarity-based grouping strategies, leaving open whether this is optimal or just better than naive pruning.
Research Context
This work directly builds on Router-weighted Expert Activation Pruning (REAP) from prior research on MoE compression, shifting the paradigm from deletion to consolidation. It contributes to the broader efficiency frontier for LLMs—alongside quantization, LoRA, and other compression techniques—but focuses specifically on the expert dimension of sparse models. The paper benchmarks against multiple MoE architectures (likely including models like Switch Transformers or similar), showing the approach generalizes across different expert configurations. This opens a research direction toward learned merging strategies that could adaptively choose compression ratios per task or even per inference batch, beyond the static per-expert merging REAM demonstrates.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::