LoRA: Low-Rank Adaptation - Interactive Visualization

LoRA fine-tunes LLMs by decomposing weight updates into two small matrices A and B (rank r), reducing trainable parameters by 100–10,000×. Instead of updating all of W, it learns ΔW = BA where B is d×r and A is r×d. This demo shows the decomposition and the parameter savings at each rank.

• See the ΔW = BA decomposition: how a large d×d weight update is expressed as two thin matrices
• Adjust rank r and instantly see the parameter count reduction vs a full fine-tune
• Compare trainable parameter counts: rank 4, 8, 16, 64 vs full fine-tuning of a 7B model
• Understand why lower rank means fewer parameters but also a lower-dimensional update space
• See how QLoRA stacks 4-bit quantization on top of LoRA to enable fine-tuning on a single GPU

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.