QLoRA = NF4(frozen W) + FP16(LoRA A, B) + Paged Optimizers
Weight representation: 7B model
Base model (frozen) (4-bit)
+16 more
+
LoRA adapter A (16-bit)
×
LoRA adapter B (16-bit)
4-bit NF4 cells are much smaller than 16-bit ones. Only adapters A and B (tiny) are trained in fp16.
GPU memory comparison - 7B model, r=16
Full Fine-tuning98.0 GB
fp16 weights + fp32 optimizer states + gradients
LoRA (fp16 base)14.1 GB
fp16 frozen base + fp16 adapter + optimizer on adapter only
QLoRA (4-bit + r=16)3.6 GB
NF4 base frozen + fp16 LoRA adapter on top
QLoRA + Double Quant3.5 GB
Also quantizes the quantization constants (saves ~3% more)
Double Quantization
Weights (NF4)
4-bit quantized
→
Quant Constants (fp32)
Also quantized to 8-bit
→
Savings
~0.37 bits/param extra
Controls
Model size
LoRA rank r16
464
QLoRA (Dettmers et al., 2023) quantizes the frozen base model to 4-bit NF4. Only the tiny LoRA adapters (A, B) are trained in fp16. Enables 65B fine-tuning on a single A100.
NF4 vs INT4: NF4 (Normal Float 4) has optimal quantization bins for normally-distributed weights. INT4 has uniform bins. NF4 matches fp16 quality on most tasks.
QLoRA (Dettmers et al., 2023) combines 4-bit NF4 quantization of the frozen base model with fp16 LoRA adapters, enabling fine-tuning of 65B parameter models on a single A100 GPU. Double quantization further reduces memory by also quantizing the quantization constants. Only the tiny LoRA adapter matrices (A and B) are trained, while the quantized base weights remain frozen.
GPU memory comparison bars for Full FT vs LoRA vs QLoRA across 7B, 13B, and 70B models
Visual weight representation showing 4-bit NF4 base model cells vs 16-bit LoRA adapter cells
Adjustable LoRA rank slider showing how adapter size affects memory and parameter count
Double quantization explainer: how quantizing the quantization constants saves 0.37 bits per parameter
Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.
