RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models
| Authors | Yusuf Çelebi et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.19321 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Fine-tuning Large Language Models (LLMs) remains structurally uncertain despite parameter-efficient methods such as Low-Rank Adaptation (LoRA), as the layer-specific roles of internal representations are poorly understood, leading to heuristic decisions about where adaptation should be applied. We model the evolution of hidden states as a high-dimensional geometric trajectory and propose using the Ramer-Douglas-Peucker (RDP) algorithm, a parameter-free and training-free polygon simplification method that preserves global structural transitions while eliminating locally redundant changes, to identify critical breakpoints along the representation path. Crucially, we use these geometric pivots not merely for analysis, but as a direct decision signal for determining which layers should be adapted during parameter-efficient fine-tuning. By integrating this geometry-aware layer selection strategy into LoRA fine-tuning of Qwen3-8B-Base, we achieve superior performance on MMLU-Math using only 13 RDP-selected layers (81.67%), significantly outperforming both full 36-layer adaptation (79.32%) and random 13-layer selection (75.56%), as well as the baseline Qwen3-8B-Base model (74.25%). These results demonstrate that leveraging the intrinsic geometry of representation trajectories provides a robust, interpretable, and training-free signal for optimizing layer selection during model adaptation.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in fine-tuning large language models: we don't know which layers should be adapted when using parameter-efficient methods like LoRA. The authors model the evolution of hidden states across layers as geometric trajectories and apply the Ramer-Douglas-Peucker algorithm—a classical polygon simplification technique—to identify critical structural breakpoints in these representations. Instead of using RDP just for analysis, they use the detected geometric pivots as a direct signal to decide which layers to adapt during fine-tuning, eliminating the guesswork that currently dominates layer selection decisions.
Core Technical Contribution
The key innovation is treating layer-wise representation evolution as a geometric optimization problem and using RDP algorithm outputs as an automated, training-free mechanism for layer selection in parameter-efficient fine-tuning. Prior work either applies LoRA uniformly across layers, uses heuristic rules based on layer position, or requires expensive sensitivity analysis. This paper's contribution is fundamentally different: it's parameter-free (RDP has no hyperparameters to tune), training-free (no forward passes needed to identify critical layers), and directly interpretable—the algorithm identifies exactly where representation transformations shift significantly, which should correspond to where adaptation matters most. This bridges the gap between understanding internal model structure and making concrete architectural decisions for efficient adaptation.
How It Works
The pipeline operates in three stages. First, the method passes a set of input examples through the base LLM and captures hidden states (activations) at every layer, creating a sequence of high-dimensional vectors representing the model's internal representations as it processes information from input to output. Second, these state sequences are treated as trajectories in high-dimensional space; RDP algorithm simplifies these trajectories by identifying critical waypoints—positions where the representation makes significant geometric turns or transitions—while removing locally redundant points that don't contribute to overall structural change. Third, the detected critical breakpoints (layers where RDP identifies major transitions) are flagged as layers where adaptation should be applied; LoRA modules are then inserted only at these selected layers rather than uniformly. The beauty of RDP is that it's purely geometric and doesn't require any training or task-specific tuning—it simply identifies structural importance based on how much the representation space itself is changing.
Production Impact
For engineering teams, this approach directly reduces the combinatorial problem of deciding which layers to adapt in parameter-efficient fine-tuning from exhaustive search or intuition-based guessing to an automated, deterministic procedure. In practice, this means faster experimentation cycles: you can identify optimal layer configurations without running multiple expensive fine-tuning trials, reducing wall-clock time and compute cost significantly. The method is particularly valuable when fine-tuning on diverse downstream tasks where layer importance varies—you can run RDP analysis once on representative input data to guide LoRA placement, then reuse that configuration across similar tasks. However, there are trade-offs: the RDP analysis itself requires a forward pass through the base model on representative data (compute cost), and the method assumes that geometric transitions in hidden states correlate with functional importance—a reasonable but unproven assumption that may not hold for all model architectures or task distributions.
Limitations and When Not to Use This
The paper's core assumption—that geometric breakpoints in representation trajectories correspond to functionally important layers for task-specific adaptation—is intuitive but lacks theoretical grounding and hasn't been validated across diverse model families, scales, or task types. RDP's effectiveness depends on choosing representative input data for the forward pass; if your inference distribution differs significantly from the data used for RDP analysis, layer selection may be suboptimal. The method also assumes relatively stable hidden state geometry; models with highly stochastic or chaotic internal representations might yield unreliable breakpoint detection. Additionally, the paper doesn't address how RDP layer selection interacts with other orthogonal efficiency techniques (quantization, pruning, knowledge distillation) or whether geometric criticality actually predicts adaptation effectiveness versus other signals like gradient flow or feature importance measured empirically during fine-tuning.
Research Context
This work sits at the intersection of mechanistic interpretability (understanding what different layers do) and parameter-efficient fine-tuning (LoRA and variants), building on growing evidence that not all layers contribute equally to task-specific adaptation. It improves upon prior heuristics like 'adapt all layers' or 'adapt later layers only' by proposing a principled, geometric criterion. The paper connects classical computational geometry (RDP algorithm from cartography and computer vision) to modern deep learning, which is a fresh perspective in the fine-tuning literature dominated by gradient-based or attention-based selection methods. This opens a research direction: using geometric and topological properties of learned representations to guide architectural decisions, potentially applicable beyond layer selection to questions about which neurons to prune, which attention heads matter, or how to structure mixture-of-experts routing.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::