JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
| Authors | Alexandra Dragomir et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.16171 |
| Download | |
| Categories | cs.LG, cs.AI, cs.CL |
Abstract
Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
Engineering Breakdown
Plain English
JumpLoRA addresses catastrophic forgetting in continual learning for Large Language Models by introducing sparse, gated adapters that dynamically isolate task-specific parameters. The paper proposes using JumpReLU gating to selectively activate different parts of LoRA (Low-Rank Adaptation) blocks, preventing interference between sequentially learned tasks without requiring explicit constraints between adapter matrices. The method is modular and works as a drop-in enhancement for existing LoRA-based continual learning approaches like IncLoRA, significantly improving performance while maintaining parameter efficiency.
Core Technical Contribution
The core novelty is applying JumpReLU gating—a dynamic activation function that produces discrete, sparse outputs—to LoRA adapter blocks for the first time in continual learning. Rather than imposing hard constraints on adapter overlap (as prior work does via subspace or coordinate-wise targeting), JumpLoRA achieves parameter isolation through learned, adaptive sparsity patterns that naturally partition the parameter space across tasks. This soft constraint approach is more flexible than prior methods and requires no task-specific hyperparameter tuning, making it genuinely modular and compatible with any LoRA-based CL framework.
How It Works
JumpLoRA inserts JumpReLU gating modules within the standard LoRA architecture, specifically gating the outputs of the low-rank decomposed matrices (down-projection and up-projection). During training on a new task, the gating mechanism learns which neurons/dimensions to activate, creating a sparse binary mask that effectively partitions the adapter parameter space. The key insight is that JumpReLU (a rectified linear unit variant that outputs discrete values) encourages binary activation patterns, so different tasks naturally learn to use disjoint subsets of adapter capacity. At inference time, only the activated dimensions contribute to the forward pass, reducing both memory and computation while maintaining task-specific learned representations.
Production Impact
For teams deploying continual learning systems on LLMs, JumpLoRA offers a straightforward way to add new tasks without retraining existing adapters or managing complex constraint matrices. The sparsity directly reduces inference latency and memory footprint per task—especially valuable when serving dozens of fine-tuned task adapters in parallel on resource-constrained hardware (mobile, edge, API serving). Integration is low-friction since it wraps existing LoRA implementations; you replace standard LoRA blocks with gated versions and retrain each new task normally. The trade-off is that JumpReLU gating adds a small amount of learnable parameters and computational overhead during training, but the resulting sparse adapters are typically 20-40% smaller in effective active parameters than dense adapters.
Limitations and When Not to Use This
The paper does not address what happens when task distributions shift significantly or when tasks have substantial overlap in the feature space—sparse partitioning may fail if tasks genuinely require similar representations. JumpReLU gating assumes that discrete, binary activation patterns are optimal for task isolation, but this may not hold for tasks that require smooth, compositional learning across task boundaries. The approach also requires careful initialization and tuning of the gating mechanism's learning rate and regularization strength; the paper does not fully characterize sensitivity to these hyperparameters. Additionally, there is no analysis of how JumpLoRA scales to hundreds of sequential tasks or very long-context scenarios where adapter overhead becomes the dominant bottleneck.
Research Context
JumpLoRA builds on the established literature of adapter-based continual learning (IncLoRA, others) and low-rank adaptation (LoRA itself, originally from Hu et al., 2021), adding a gating mechanism inspired by recent work on sparse neural networks and dynamic sparsity. The work contributes to the broader challenge of scaling LLM fine-tuning without catastrophic forgetting—a critical problem as foundation models are increasingly adapted to multiple downstream tasks. This research opens a direction toward dynamic, learned sparsity patterns as an alternative to explicit constraint-based approaches, potentially inspiring follow-up work on adaptive parameter routing and task-specific gating in other adaptation paradigms (prefix tuning, LoTA, etc.).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::