Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
| Authors | Yixuan Tang & Yi Yang |
| Year | 2026 |
| HF Upvotes | 18 |
| arXiv | 2604.16826 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update ΔW = BA as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix B. Across tasks, B repeatedly uses a small set of shared directions, while A remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates B before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.
Engineering Breakdown
Plain English
This paper tackles the problem of merging multiple LoRA adapters trained separately on different tasks, which usually degrades performance compared to training jointly. The authors discovered that the core issue is not symmetric: the output-side matrix B in LoRA (where ΔW = BA) repeatedly reuses a small set of shared directions across tasks, while the input-side matrix A stays task-specific. This causes the merged adapter to over-amplify these shared directions and lose task-specific information. They propose Pico, a data-free method that downscales the over-shared directions in B before merging and then rescales the final merged update, which integrates directly into existing merging workflows without requiring additional training data.
Core Technical Contribution
The core novelty is asymmetric analysis of LoRA matrix interference during merging—showing that B (not A) is the bottleneck. Prior work treated ΔW = BA as an indivisible unit and applied symmetric merging strategies (e.g., averaging weights directly). This paper reveals that B exhibits severe direction reuse across tasks (shared subspace crowding), while A remains diverse. The algorithmic contribution is Pico: a calibration-based approach that explicitly downscales over-shared B directions using spectral analysis, then rescales the merged adapter to restore magnitude. This is fundamentally different because it recognizes the asymmetry and targets the actual source of interference rather than treating both matrices equally.
How It Works
Pico operates in three stages. First, it analyzes the LoRA adapters from multiple tasks and computes the spectral properties of the B matrices to identify shared directions—directions that are used across many tasks with high magnitude. Second, it downscales these shared directions in B before merging by reducing their singular values or applying direction-specific scaling factors (essentially zeroing out or dampening the crowded subspace). Third, after merging (e.g., via weighted averaging), it rescales the entire merged update by a learnable or data-driven factor to restore the magnitude lost during downscaling. The process is data-free: it does not require labeled examples and works purely from the geometry of the pre-trained LoRA matrices themselves. The merged adapter then replaces the original single-task adapter at inference time.
Production Impact
For teams deploying multi-task models via LoRA merging (common in fine-tuning scenarios), this addresses a critical gap: merged LoRA adapters often underperform individual ones, forcing painful choices between model size and task coverage. Pico is a drop-in post-processing step that improves merged adapter quality without retraining or data requirements—you extract LoRA matrices, apply the calibration, and merge normally. The computational overhead is minimal (spectral analysis on small adapter matrices is cheap). However, integration requires storing per-adapter metadata (spectral information) and understanding which directions are shared—this adds bookkeeping complexity. The main trade-off is that calibration assumes tasks share some common representation (if tasks are completely orthogonal, Pico provides minimal benefit). For production systems managing 5+ task adapters, the performance gain likely justifies the modest integration complexity.
Limitations and When Not to Use This
The paper assumes shared directions in B are actually problematic—this may not hold if the shared subspace genuinely benefits all tasks or if tasks naturally cluster into groups with different shared subspaces. It is data-free, which is a strength for privacy but also means it cannot leverage task-specific validation data to tune the downscaling factor more accurately. The method's effectiveness likely depends on LoRA rank and the degree of task similarity; extremely low-rank or highly orthogonal task sets may not exhibit the crowding phenomenon or may not benefit from direction calibration. The paper does not deeply explore failure modes: what happens when downscaling shared directions removes genuinely useful cross-task signal? Finally, it is unclear how Pico scales to very large numbers of adapters (10+) or extreme rank imbalances across tasks—the shared direction identification may become unstable.
Research Context
This work sits at the intersection of parameter-efficient fine-tuning (LoRA, introduced by Hu et al. 2021) and multi-task/adapter merging (a growing area with works on averaged LoRA, simple averaging, and task arithmetic). It extends the understanding of why naive merging fails, moving beyond empirical observations to geometric root causes. The paper likely benchmarks on multi-task datasets like MTL benchmarks or domain adaptation sets. It opens a direction toward asymmetric analysis of other low-rank adapter techniques (prefix tuning, adapters, etc.) and raises the question of whether similar crowding occurs in other matrix factorizations. This could inspire future work on task-aware LoRA design or online calibration during merging.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::