UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 7 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Minbin Huang et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2605.06665 |
| Download | |
| Code | https://github.com/Centaurus-Alpha/UniPool |
Abstract
Modern Mixture-of-Experts (MoE) architectures allocate expert capacity through a rigid per-layer rule: each transformer layer owns a separate expert set. This convention couples depth scaling with linear expert-parameter growth and assumes that every layer needs isolated expert capacity. However, recent analyses and our routing probe challenge this allocation rule: replacing a deeper layer's learned top-k router with uniform random routing drops downstream accuracy by only 1.0-1.6 points across multiple production MoE models. Motivated by this redundancy, we propose UniPool, an MoE architecture that treats expert capacity as a global architectural budget by replacing per-layer expert ownership with a single shared pool accessed by independent per-layer routers. To enable stable and balanced training under sharing, we introduce a pool-level auxiliary loss that balances expert utilization across the entire pool, and adopt NormRouter to provide sparse and scale-stable routing into the shared expert pool. Across five LLaMA-architecture model scales (182M, 469M, 650M, 830M, and 978M parameters) trained on 30B tokens from the Pile, UniPool consistently improves validation loss and perplexity over the matched vanilla MoE baselines. Across these scales, UniPool reduces validation loss by up to 0.0386 relative to vanilla MoE. Beyond raw loss improvement, our results identify pool size as an explicit depth-scaling hyperparameter: reduced-pool UniPool variants using only 41.6%-66.7% of the vanilla expert-parameter budget match or outperform layer-wise MoE at the tested scales. This shows that, under a shared-pool design, expert parameters need not grow linearly with depth; they can grow sublinearly while remaining more efficient and effective than vanilla MoE. Further analysis shows that UniPool's benefits compose with finer-grained expert decomposition.
Engineering Breakdown
Plain English
UniPool challenges the standard Mixture-of-Experts design where each transformer layer gets its own isolated set of experts. The authors found that replacing a layer's router with random routing only drops accuracy by 1-1.6%, suggesting layers don't actually need dedicated expert pools. They propose instead using a single shared expert pool that all layers access through independent routers, treating expert capacity as a global budget rather than a per-layer resource.
Key Engineering Insight
Expert redundancy across layers is much higher than assumed—most layers can route effectively to the same shared pool without significant accuracy loss. This means you can decouple model depth from total expert parameter count, fundamentally changing how you scale MoE models.
Why It Matters for Engineers
If UniPool maintains performance with fewer total experts, it directly reduces memory footprint and compute cost for MoE inference at production scale. This matters because current MoE models grow expert parameters linearly with depth, making them expensive to serve; a shared pool could let you add layers without proportional parameter growth.
Research Context
Prior MoE work assumed per-layer expert isolation was necessary for routing specialization and model quality. UniPool advances beyond this by proving layers can share capacity while maintaining independent routing decisions, opening the door to more efficient MoE scaling strategies that decouple architectural depth from parameter multiplication.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::