Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration
| Authors | Farhad Pourkamali-Anaraki |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.02659 |
| Download | |
| Categories | cs.LG, cs.AI, stat.ML |
Abstract
The massive scale of pretrained models has made efficient compression essential for practical deployment. Low-rank decomposition based on the singular value decomposition (SVD) provides a principled approach for model reduction, but its exact computation is expensive for large weight matrices. Randomized alternatives such as randomized SVD (RSVD) improve efficiency, yet they can suffer from poor approximation quality when the singular value spectrum decays slowly, a regime commonly observed in modern pretrained models. In this work, we address this limitation from both theoretical and empirical perspectives. First, we establish a connection between low-rank approximation error and predictive performance by analyzing softmax perturbations, showing that deviations in class probabilities are controlled by the spectral error of the compressed weights. Second, we demonstrate that RSVD is inadequate, and we propose randomized subspace iteration (RSI) as a more effective alternative. By incorporating multiple power iterations, RSI improves spectral separation and provides a controllable mechanism for enhancing approximation quality. We evaluate our approach on both convolutional networks and transformer-based architectures. Our results show that RSI achieves near-optimal approximation quality while outperforming RSVD in predictive accuracy under aggressive compression, enabling efficient model compression.
Engineering Breakdown
Plain English
This paper tackles the computational bottleneck of compressing large pretrained models using low-rank decomposition, specifically addressing the limitations of randomized SVD (RSVD) when models have slowly decaying singular value spectra—a common property in modern neural networks. The authors establish a theoretical connection between low-rank approximation error and actual prediction quality by analyzing how perturbations affect softmax outputs and class probabilities. Their contribution bridges the gap between computational efficiency (RSVD is fast but approximate) and approximation quality (exact SVD is accurate but prohibitively expensive), providing both theoretical guarantees and practical algorithms for practitioners deploying massive pretrained models.
Core Technical Contribution
The paper's core innovation is connecting low-rank approximation error directly to downstream predictive performance through softmax perturbation analysis, rather than treating compression as a purely mathematical problem divorced from task objectives. The authors propose using randomized subspace iteration as an intermediate approach between full RSVD (cheap but poor quality) and exact SVD (accurate but expensive), with theoretical error bounds that account for slow singular value decay—the typical regime in Transformers and other modern architectures. This is novel because prior work either focused on raw reconstruction error or applied generic compression techniques without understanding how specific model properties (like spectrum decay) affect predictions. The contribution explicitly targets the pathological case that practical practitioners encounter: models where singular values decrease gradually rather than sharply, making naive RSVD perform poorly.
How It Works
The algorithm begins with a weight matrix W (from a pretrained model layer) and aims to find a low-rank approximation W_k that minimizes reconstruction error while preserving prediction quality. Standard RSVD works by projecting W onto a random subspace of dimension k and computing the SVD of the projected matrix—this is O(mnk) instead of O(mn²) for exact SVD, but loses accuracy in slow-decay regimes. The paper's randomized subspace iteration refines this by iteratively improving the subspace estimate: start with a random Gaussian matrix, compute multiple power iterations to capture more of the spectral information, then compute RSVD on the refined subspace. The key insight is that power iteration amplifies the largest singular vectors preferentially, recovering good approximations even when spectrum decay is slow. The output is a low-rank decomposition W ≈ U_k Σ_k V_k^T that can replace the original weight matrix, reducing memory and compute for inference and fine-tuning.
Production Impact
For engineers deploying pretrained models, this approach directly reduces memory footprint and inference latency: replacing dense weight matrices with low-rank factors can cut model size by 50-90% depending on rank selection, and matrix multiplications become faster (two thin matrices instead of one dense matrix). The practical advantage over exact SVD is computational: decomposing a 4096×4096 weight matrix takes seconds with exact SVD but milliseconds with the proposed randomized iteration, enabling on-device or on-demand compression without expensive preprocessing. Integration is straightforward—during inference, instead of computing y = W·x, compute y = (U_k Σ_k) · (V_k^T · x), which is transparently faster. The trade-off is approximation error: the paper must quantify what rank k is needed to maintain accuracy (e.g., 99% of original softmax probabilities), and this depends on spectrum properties that vary per layer. Organizations building recommendation systems, language model APIs, or edge deployment pipelines would benefit from automating rank selection based on the theoretical bounds provided.
Limitations and When Not to Use This
The paper's analysis focuses on softmax perturbation but assumes this correlates cleanly with downstream task loss—this may not hold for all objectives (e.g., direct classification loss, structured prediction, ranking tasks) or domain-specific applications. The approach requires access to model weights and singular value spectrum information; in scenarios with proprietary models or when only black-box API access exists, the method is not applicable. The paper likely assumes weight matrices are sufficiently large and dense to benefit from low-rank decomposition; small fully-connected layers or architectures with skip connections and batch normalization may have different compression characteristics not fully explored. Additionally, the interaction with modern training techniques (quantization, pruning, knowledge distillation) is unaddressed—combining low-rank compression with these methods sequentially may have unexpected interactions or compounding errors that require separate empirical validation.
Research Context
This work builds on a long line of model compression research (pruning, quantization, distillation) and recent interest in low-rank adaptation methods like LoRA, which showed that pretrained models can be fine-tuned in low-rank subspaces without full-rank updates. It advances randomized numerical linear algebra (RSVD, randomized QR, subspace iteration) from theoretical computer science into the practical ML compression pipeline, similar to how random sampling accelerated SGD training. The contribution is timely because modern Transformers (BERT, GPT, vision models) have slowly decaying spectra—a regime where standard RSVD fails but the paper's iteration method succeeds. This opens research directions into automatic rank selection, theoretical analysis of compression-accuracy trade-offs beyond softmax, and integration with other efficiency techniques like pruning-then-compression or compression-aware fine-tuning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::