Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 1 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Meng Lou et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2602.03473 |
| Download | |
| Code | https://github.com/LMMMEng/CaRE |
Abstract
Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose CaRE, a scalable {C}ontinual Le{a}rner with efficient Bi-Level {R}outing Mixture-of-{E}xperts (BR-MoE). The core idea of BR-MoE is a bi-level routing mechanism: a router selection stage that dynamically activates relevant task-specific routers, followed by an expert routing phase that dynamically activates and aggregates experts, aiming to inject discriminative and comprehensive representations into every intermediate network layer. On the other hand, we introduce a challenging dataset, OmniBenchmark-1K, for CIL performance evaluation on very long task sequences with hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of datasets and task settings, including commonly used CIL datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. We hope that this work will inspire further research into continual learning over extremely long task sequences. Code and dataset are publicly released at https://github.com/LMMMEng/CaRE.
Engineering Breakdown
Plain English
This paper introduces CaRE, a continual learning system that uses a two-level routing mechanism (bi-level routing Mixture-of-Experts) to train on 300+ sequential tasks without forgetting previous knowledge. The key innovation is dynamically activating task-specific routers and experts rather than using the entire model, which improves both accuracy and computational efficiency while maintaining stability as new tasks arrive.
Key Engineering Insight
The bi-level routing approach separates task selection from expert selection—first determining which task routers are relevant, then routing to specific experts within those routers. This hierarchical gating is more efficient and stable than flat MoE routing for long continual learning sequences, directly addressing the stability-plasticity tradeoff that kills naive fine-tuning.
Why It Matters for Engineers
Production systems often need to learn from streaming data without retraining from scratch (e.g., recommendation systems, fraud detection, autonomous agents). This paper demonstrates a scalable method to handle 300+ distinct tasks without catastrophic forgetting—a fundamental blocker for deploying continual learning in real products. The routing efficiency also reduces inference compute, which matters for deployed models.
Research Context
Prior continual learning work struggled with either forgetting old tasks or being too computationally expensive for long task sequences. This paper advances the field by showing MoE architectures with intelligent routing can scale to practical lengths (300+ tasks) when combined with pre-trained models. It enables the next generation of systems that learn continuously from user interactions without the overhead of periodic retraining or replay buffers.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::