Advanced PEFT Methods

The Production Wake-Up Call

It was 3 AM when the alert fired. A startup had just shipped a multilingual customer support assistant built on a 13B parameter base model. They had fine-tuned it with full LoRA - rank 64, all attention layers, 8-bit quantization - and it worked beautifully in staging. Then they tried to serve five language-specific variants simultaneously on a single A100 80GB instance to cut costs.

The math was brutal. Each LoRA adapter was 120MB. Five adapters meant 600MB just for the delta weights, loaded and hot-swapped per request. Latency spiked on adapter switches. Memory fragmentation made the GPU grumpy. The on-call engineer stared at the dashboard and thought: there has to be a better way.

There is. The field of parameter-efficient fine-tuning did not stop at LoRA. Between 2021 and 2024, researchers published a family of methods that push the boundary further - methods that use 10x fewer parameters than LoRA, methods that adapt more intelligently by learning which layers matter most, methods that survive quantization without quality loss, and methods that share weights across layers so dozens of task-specific variants cost almost nothing to store.

This lesson maps that landscape. You will understand not just what each method does, but why it was invented - what specific failure mode of the previous approach it was designed to fix. By the end, you will be able to look at a fine-tuning problem and pick the right tool: not always LoRA, sometimes something sharper.

The production engineer does not need to master every method. They need to know the decision tree: when IA3 is enough, when AdaLoRA is worth the overhead, when LoftQ saves the day on a consumer GPU. That decision tree is what this lesson builds.

---

Why This Exists - The Problem LoRA Did Not Fully Solve

LoRA solved the catastrophic forgetting and full fine-tuning cost problems elegantly. But as teams deployed it at scale, three new pain points emerged.

The parameter count problem. LoRA at rank 8 on a 7B model is roughly 20-40M trainable parameters. That sounds small until you need 50 task-specific variants. Storage costs scale linearly. Loading time matters. If your inference server needs to hot-swap adapters per request, even 40MB per adapter adds up.

The rank allocation problem. Standard LoRA uses the same rank for every layer. But not every layer is equally important for a given task. The early layers of a transformer capture syntactic and factual knowledge. The later layers handle task-specific reasoning. Giving rank 8 to a layer that barely changes during fine-tuning wastes parameters. Giving rank 8 to a layer that needs rank 32 to capture the task signal underfits. A fixed rank budget is a blunt instrument.

The quantization initialization problem. When you quantize a base model to 4-bit before adding LoRA adapters (QLoRA), the quantization error gets baked in. The LoRA adapters then have to compensate for both the task signal and the quantization error. If the initialization is poorly chosen, the adapter starts from a worse starting point and needs more steps to converge. For very low-bit quantization (2-bit, 3-bit), this compounds badly.

Each advanced PEFT method in this lesson was invented to fix one of these problems - or to push the efficiency boundary even further in scenarios where LoRA is more than you need.

---

Historical Context - A Field That Moved Fast

The timeline of PEFT methods compresses four years of research into a surprisingly coherent narrative.

2021: The Prompt Era. Two papers from different teams arrived almost simultaneously and set the stage. Li and Liang at Stanford published Prefix Tuning in January 2021, showing that prepending learnable "virtual token" representations to each transformer layer could adapt GPT-2 and BART without touching any base weights. Lester, Al-Rfou, and Constant at Google published Prompt Tuning in September 2021 - a simpler variant that only added soft tokens at the input layer. Both methods came from the same insight: if attention can attend to anything, give it something learnable to attend to.

2022: LoRA Takes Over. Hu et al. published LoRA at ICLR 2022, and it dominated. The low-rank matrix decomposition idea was cleaner and more flexible than prefix manipulation. But the Prefix Tuning and Prompt Tuning papers left a legacy: they proved that tiny interventions could drive large behavioral changes.

2022-2023: The Efficiency Race. With LoRA established as the baseline, researchers started asking how much further parameters could be cut. Liu et al. published IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations) in 2022. Instead of adding matrices, IA3 learns simple scaling vectors for key, value, and feed-forward activations - 100x fewer parameters than LoRA. Zhang et al. published AdaLoRA the same year, introducing singular value decomposition to allocate the rank budget adaptively based on importance scores computed during training.

2023: Quantization-Aware Methods. As QLoRA popularized 4-bit fine-tuning, a new class of problems emerged around initialization quality. Kopiczko et al. published VeRA (Vector-based Random Matrix Adaptation) in late 2023, showing that if you share frozen random matrices across all layers and only learn tiny scaling vectors, you could match LoRA quality with a fraction of the parameters. Simultaneously, Guo et al. published LoftQ (LoRA-Fine-Tuning-aware Quantization), which alternates between quantizing the base weights and finding LoRA initialization that minimizes the resulting approximation error.

The "aha moment" for the whole field came from an unexpected direction. Aghajanyan et al. (2020) had shown empirically that pre-trained representations have a very low "intrinsic dimensionality" - the number of parameters actually needed to solve a downstream task is much smaller than the model size. PEFT methods are different computational paths to exploit that same insight.

---

Core Concepts

Prefix Tuning - Virtual Tokens in Every Layer

The intuition for Prefix Tuning comes from how prompts work in natural language. If you prepend "Translate the following English text to French:" to any input, the model's attention mechanism is guided toward translation behavior. The prefix creates a context that steers all downstream processing.

Prefix Tuning takes this idea and makes the prefix learnable - and importantly, puts a learnable prefix at every transformer layer, not just the input.

In a standard transformer, each attention layer computes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

where $Q$, $K$, $V$ are derived from the sequence tokens. In Prefix Tuning, learnable prefix matrices $P_K$ and $P_V$ are prepended to the keys and values:

$K' = [P_K; K], \quad V' = [P_V; V]$

The attention now has access to these virtual tokens at every layer. The parameters are only the prefix matrices - base model weights are frozen. For a model with $L$ layers and prefix length $l$, the total new parameters are $2 \times L \times l \times d_{model}$.

Li and Liang found that directly optimizing the prefix embeddings was unstable. They used a reparameterization trick: a small MLP maps a lower-dimensional vector to the full prefix, and only the MLP is trained. At inference time, the MLP is discarded and the computed prefix is used directly.

from peft import PrefixTuningConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "facebook/opt-1.3b"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Configure prefix tuning
peft_config = PrefixTuningConfig(
    task_type="CAUSAL_LM",
    num_virtual_tokens=20,        # prefix length per layer
    encoder_hidden_size=128,      # MLP hidden size for reparameterization
    prefix_projection=True,       # use MLP reparameterization
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# trainable params: 983,040 || all params: 1,316,634,624 || trainable%: 0.0747

When to use Prefix Tuning: It works best for sequence-to-sequence tasks (summarization, translation) where the "context" metaphor maps cleanly onto the task. It performs poorly on classification tasks where the decision boundary is complex. The per-layer prefix means it interacts with intermediate representations more deeply than input-only methods.

---

Prompt Tuning - Simpler, Input Only

Prompt Tuning (Lester et al. 2021) is the minimalist version of Prefix Tuning. Instead of learnable tokens at every layer, only the input embedding layer gets learnable soft tokens prepended.

The model sees: $[v_1, v_2, ..., v_k, x_1, x_2, ..., x_n]$ where $v_i$ are the learnable soft prompt tokens and $x_i$ are the real input tokens.

The total new parameters are just $k \times d_{embed}$ - for 20 tokens on a model with 4096-dimensional embeddings, that is 81,920 parameters. Remarkably small.

Lester et al. showed something important: at very large model scales (11B parameters), Prompt Tuning nearly matches full fine-tuning performance. At smaller scales, the gap widens. This "scale sensitivity" is a key limitation.

from peft import PromptTuningConfig, PromptTuningInit, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b")

peft_config = PromptTuningConfig(
    task_type="CAUSAL_LM",
    prompt_tuning_init=PromptTuningInit.TEXT,
    num_virtual_tokens=20,
    prompt_tuning_init_text="Classify the sentiment of this review:",
    tokenizer_name_or_path="facebook/opt-1.3b",
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# trainable params: 81,920 || all params: 1,316,716,544 || trainable%: 0.0062

Initialization matters. Random initialization works, but initializing from a text prompt that describes the task converges faster and often reaches higher final quality. The PromptTuningInit.TEXT option handles this.

When to use Prompt Tuning: When you need the absolute minimum parameter count, when you are working with very large base models (13B+), when serving many task variants from one base model instance (soft prompts are trivially small to swap), or when you want a baseline before trying LoRA.

---

IA3 - Scale, Don't Add

IA3 (Liu et al. 2022) came from a different question: instead of adding parameters, what if you scaled existing activations?

The insight is that transformer computations involve a series of activations that modulate information flow. If you multiply these activations by learned scaling vectors, you can reshape what the model pays attention to without changing the weight matrices at all.

IA3 learns three sets of scaling vectors per transformer layer:

• $l_k \in \mathbb{R}^{d_k}$ - scales the key activations in attention
• $l_v \in \mathbb{R}^{d_v}$ - scales the value activations in attention
• $l_{ff} \in \mathbb{R}^{d_{ff}}$ - scales the inner feed-forward activations

The modified attention is:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q(l_k \odot K)^T}{\sqrt{d_k}}\right)(l_v \odot V)$

where $\odot$ is element-wise multiplication. The feed-forward modification is:

$\text{FFN}(x) = W_2 \cdot (l_{ff} \odot \gamma(W_1 x))$

where $\gamma$ is the activation function (GELU, SiLU, etc.).

The parameter count is tiny. For a model with $L$ layers, $d_k$ key dimension, $d_v$ value dimension, and $d_{ff}$ feed-forward inner dimension, total IA3 parameters = $L \times (d_k + d_v + d_{ff})$. For Llama-3 8B, this is roughly 800K parameters - compared to LoRA rank-8 at roughly 20M.

from peft import IA3Config, get_peft_model, TaskType
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-large")

ia3_config = IA3Config(
    task_type=TaskType.SEQ_2_SEQ_LM,
    target_modules=["k", "v", "wo"],   # keys, values, feed-forward output
    feedforward_modules=["wo"],         # which of target_modules are FF layers
)

model = get_peft_model(model, ia3_config)
model.print_trainable_parameters()
# trainable params: 282,624 || all params: 783,144,960 || trainable%: 0.0361

# Training loop - exactly the same as LoRA
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./ia3-flan-t5",
    num_train_epochs=5,
    per_device_train_batch_size=16,    # larger batch because parameters are tiny
    learning_rate=3e-3,                # IA3 needs higher LR than LoRA
    fp16=True,
    logging_steps=50,
    save_strategy="epoch",
)

IA3 needs a higher learning rate than LoRA. Because the scaling vectors start at 1.0 (no change) and need to move meaningfully, learning rates around 3e-3 to 1e-2 work better than the 1e-4 to 3e-4 typical for LoRA.

IA3 merges for free. Because the modification is element-wise scaling, you can fold the learned vectors directly into the weight matrices at inference time - the same way LoRA adapters can be merged. The result is a model with zero inference overhead.

# Merge IA3 vectors into base model weights - zero inference overhead
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./ia3-merged")

When to use IA3: Few-shot adaptation tasks, classification and information extraction, scenarios where you need dozens of task variants with minimal storage, and as a first experiment before deciding whether you need the expressiveness of LoRA. It does not work as well for generative tasks that require large behavioral shifts (style transfer, code generation from scratch).

---

AdaLoRA - Adaptive Rank Allocation

Standard LoRA assigns the same rank to every layer. AdaLoRA (Zhang et al. 2022) asks: what if layers get different ranks based on how much they need to change?

The mathematical foundation is singular value decomposition (SVD). Any matrix $W$ can be decomposed as $W = U \Sigma V^T$ where $\Sigma$ is diagonal with singular values ordered by magnitude. The rank of $W$ is the number of non-zero singular values. Truncating to the top-$r$ singular values gives the best rank-$r$ approximation.

AdaLoRA parameterizes the weight update as:

$\Delta W = P \Lambda Q^T$

where $P$ and $Q$ are orthogonal matrices and $\Lambda = \text{diag}(\lambda_1, ..., \lambda_r)$ is a diagonal matrix of singular values. During training, AdaLoRA computes an importance score for each singular value based on the product of its magnitude and gradient sensitivity. Singular values with low importance scores are pruned - effectively reducing the rank of that layer's adapter.

The total rank budget is fixed (say, total rank = 512 across 32 layers). AdaLoRA redistributes this budget: layers that need more expressiveness grow to rank 32, layers that barely change shrink to rank 2 or 0.

from peft import AdaLoraConfig, get_peft_model, TaskType
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "roberta-large",
    num_labels=2
)

adalora_config = AdaLoraConfig(
    task_type=TaskType.SEQ_CLS,
    init_r=12,                   # initial rank for all layers
    target_r=4,                  # target average rank after pruning
    beta1=0.85,                  # EMA coefficient for importance score
    beta2=0.85,
    tinit=200,                   # steps before rank pruning starts
    tfinal=1000,                 # steps when rank budget is finalized
    deltaT=10,                   # rank reallocation interval (steps)
    lora_alpha=32,
    target_modules=["query", "value", "key", "dense"],
)

model = get_peft_model(model, adalora_config)
model.print_trainable_parameters()

# AdaLoRA requires its own update step inside the training loop
# Use the Trainer API - it handles this automatically
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./adalora-roberta",
    num_train_epochs=10,
    per_device_train_batch_size=32,
    learning_rate=3e-4,
    warmup_ratio=0.06,
    weight_decay=0.1,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

# Important: AdaLoRA rank updates happen inside the Trainer
# Do not use a custom training loop without calling model.update_and_allocate()
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.train()

The rank schedule matters. tinit controls how many warm-up steps happen before pruning begins. Starting pruning too early (before the model has learned what each layer does) leads to poor rank allocation. A rule of thumb: tinit should be 5-10% of total training steps.

When to use AdaLoRA: When you have budget for slightly more complex training but want better quality-per-parameter than fixed LoRA. Particularly strong on NLU benchmarks (GLUE, SuperGLUE), question answering, and any task where different layers have obviously different roles. Not recommended when training speed is the primary concern - the rank management overhead adds 10-15% to training time.

---

VeRA - Shared Random Projections

VeRA (Kopiczko et al. 2023) pushes the efficiency argument to its logical extreme. The observation: LoRA uses different $A$ and $B$ matrices for each layer. But the information content in those matrices might be highly redundant across layers. What if all layers shared the same random projection matrices?

VeRA freezes a single pair of random matrices $A_{shared}$ and $B_{shared}$ (initialized once, never trained) and learns only small scaling vectors $b$ and $d$ per layer:

$\Delta W_l = \text{diag}(d_l) \cdot B_{shared} \cdot \text{diag}(b_l) \cdot A_{shared}$

where $b_l \in \mathbb{R}^r$ and $d_l \in \mathbb{R}^{d_{out}}$ are the only learned parameters for layer $l$.

The total trainable parameters collapse to $L \times (r + d_{out})$ instead of $L \times r \times (d_{in} + d_{out})$. For Llama-2 7B with rank 64, LoRA uses roughly 160M parameters. VeRA uses roughly 2.6M - a 60x reduction.

from peft import VeraConfig, get_peft_model, TaskType

# VeRA is available in PEFT >= 0.9.0
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

vera_config = VeraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=256,                       # VeRA needs higher r than LoRA to compensate
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    vera_dropout=0.1,
    projection_prng_key=42,      # seed for random projection matrices
    save_projection=True,        # save shared matrices with adapter
    d_initial=0.1,               # initial value for scaling vector d
)

model = get_peft_model(model, vera_config)
model.print_trainable_parameters()
# trainable params: 2,638,848 || all params: 6,740,148,224 || trainable%: 0.039

VeRA requires a higher rank. Because the shared random matrices are fixed, they cannot be optimized to align with the task. Compensating for this requires more dimensions - typical VeRA configs use $r = 256$ or $r = 512$, much higher than LoRA's typical 8-64. The scaling vectors then learn to select the useful directions from this large random projection.

When to use VeRA: When you need to train many task-specific adapters and storage is the bottleneck. All adapters share the same frozen random matrices (store once), so incremental storage per task is just two small scaling vectors. Excellent for continual learning scenarios and multi-tenant serving.

---

LoftQ - Quantization-Aware Initialization

The problem LoftQ (Guo et al. 2023) solves is subtle but important. In QLoRA, the workflow is:

1. Quantize the base model to 4-bit (or lower)
2. Initialize LoRA adapters randomly
3. Train

The issue: quantization introduces error $E = W - Q(W)$ where $Q$ is the quantization function. The LoRA adapters must now compensate for both the task signal and the quantization error. If the initial LoRA matrices are random, the training starts far from a good solution.

LoftQ reframes initialization as an optimization problem. Given quantized weights $Q(W)$ and LoRA matrices $A$, $B$, find the best $A$ and $B$ such that:

$\|W - Q(W) - BA\|_F \text{ is minimized}$

The solution uses alternating optimization:

1. Fix $A$ and $B$, optimize quantization: find $Q(W)$ that minimizes the residual
2. Fix quantization, optimize LoRA matrices: SVD of the residual $W - Q(W) = U\Sigma V^T$, set $B = U_r\Sigma_r^{1/2}$ and $A = \Sigma_r^{1/2}V_r^T$
3. Repeat for several iterations

# LoftQ is integrated into the PEFT library
from peft import LoftQConfig, LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
import torch

# Load base model in full precision - LoftQ quantizes internally
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16,
)

# LoftQ configuration
loftq_config = LoftQConfig(
    loftq_bits=4,            # quantization bits (4 or 2)
    loftq_iter=1,            # number of alternating optimization iterations
)

lora_config = LoraConfig(
    init_lora_weights="loftq",
    loftq_config=loftq_config,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# Training proceeds normally - but starts from a much better initialization
# Guo et al. report 1-3% better final accuracy vs random LoRA initialization
# with the same number of training steps

When to use LoftQ: When training with aggressive quantization (4-bit or lower), when you have a limited training budget (fewer steps), and when you observe that standard QLoRA adapters converge slowly or get stuck. For 8-bit quantization the benefit is smaller because quantization error is lower.

---

Architecture Overview

---

Comparison Table

Method	Params (7B model)	Relative to LoRA r=8	Best for	Weakness
Prompt Tuning	80K	0.4%	Very large models, many variants	Weak on small models
IA3	800K	4%	Classification, NLU, low storage	Less expressive for generative tasks
LoRA r=8	20M	1x baseline	General purpose	Fixed rank allocation
AdaLoRA	15-25M	0.75-1.25x	NLU benchmarks, quality-focused	Training overhead
VeRA r=256	2.6M	13%	Many task variants, continual learning	Needs high r, slower convergence
Prefix Tuning	5M	25%	Seq2seq, translation, summarization	Instability without reparameterization
LoftQ	Same as LoRA	1x + better init	4-bit quantized training	Init overhead, full precision load needed

---

IA3 vs LoRA - Practical Comparison

# Practical comparison: IA3 vs LoRA on GLUE SST-2 sentiment classification
from peft import IA3Config, LoraConfig, get_peft_model, TaskType
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
)
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score

MODEL_NAME = "roberta-base"
dataset = load_dataset("glue", "sst2")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

def tokenize(examples):
    return tokenizer(
        examples["sentence"],
        truncation=True,
        max_length=128,
        padding="max_length",
    )

tokenized = dataset.map(tokenize, batched=True)
tokenized = tokenized.rename_column("label", "labels")
tokenized.set_format("torch", columns=["input_ids", "attention_mask", "labels"])

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return {"accuracy": accuracy_score(labels, preds)}

def run_experiment(config_name, peft_config):
    model = AutoModelForSequenceClassification.from_pretrained(
        MODEL_NAME, num_labels=2
    )
    model = get_peft_model(model, peft_config)
    params = model.print_trainable_parameters()

    args = TrainingArguments(
        output_dir=f"./{config_name}",
        num_train_epochs=5,
        per_device_train_batch_size=32,
        learning_rate=3e-3 if config_name == "ia3" else 3e-4,
        evaluation_strategy="epoch",
        save_strategy="epoch",
        load_best_model_at_end=True,
        metric_for_best_model="accuracy",
        report_to="none",
    )

    trainer = Trainer(
        model=model,
        args=args,
        train_dataset=tokenized["train"],
        eval_dataset=tokenized["validation"],
        compute_metrics=compute_metrics,
    )
    trainer.train()
    results = trainer.evaluate()
    return results["eval_accuracy"]

# IA3 config
ia3_config = IA3Config(
    task_type=TaskType.SEQ_CLS,
    target_modules=["key", "value", "output.dense"],
    feedforward_modules=["output.dense"],
)

# LoRA config
lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=8,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
)

ia3_acc = run_experiment("ia3", ia3_config)
lora_acc = run_experiment("lora", lora_config)

print(f"IA3 accuracy:  {ia3_acc:.4f}")
print(f"LoRA accuracy: {lora_acc:.4f}")
# Typical results on SST-2:
# IA3 accuracy:  0.9443
# LoRA accuracy: 0.9521
# IA3 is 0.8% behind with ~25x fewer parameters

---

AdaLoRA Rank Visualization

---

Production Engineering Notes

Serving Multiple PEFT Variants

The main production advantage of ultra-lightweight methods (IA3, Prompt Tuning, VeRA) is multi-tenant serving.

# Production pattern: one base model, many task adapters hot-swapped
from peft import PeftModel
import torch

class MultiTaskServer:
    def __init__(self, base_model_path: str):
        from transformers import AutoModelForCausalLM, AutoTokenizer
        self.tokenizer = AutoTokenizer.from_pretrained(base_model_path)
        # Load base model once in shared memory
        self.base_model = AutoModelForCausalLM.from_pretrained(
            base_model_path,
            torch_dtype=torch.float16,
            device_map="cuda:0",
        )
        self.loaded_adapters = {}

    def load_adapter(self, task_name: str, adapter_path: str):
        """Load a PEFT adapter and cache it."""
        if task_name not in self.loaded_adapters:
            # For IA3/VeRA: adapter is tiny, load many into memory
            peft_model = PeftModel.from_pretrained(
                self.base_model,
                adapter_path,
                adapter_name=task_name,
            )
            self.loaded_adapters[task_name] = adapter_path
        return self

    def generate(self, task_name: str, prompt: str, **kwargs):
        """Switch adapter and generate."""
        if isinstance(self.base_model, PeftModel):
            self.base_model.set_adapter(task_name)

        inputs = self.tokenizer(prompt, return_tensors="pt").to("cuda:0")
        with torch.no_grad():
            outputs = self.base_model.generate(**inputs, **kwargs)
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

# Usage
server = MultiTaskServer("meta-llama/Llama-3.1-8B")
server.load_adapter("sentiment", "./adapters/ia3-sentiment")
server.load_adapter("ner", "./adapters/ia3-ner")
server.load_adapter("summarization", "./adapters/prefix-summarization")

result = server.generate("sentiment", "This product is amazing!")

Memory Cost Per Adapter (Approximate)

# Quick calculation: parameter count for each PEFT method on Llama-3 8B
# d_model=4096, d_ff=14336, num_layers=32, d_kv=128

configs = {
    "Prompt Tuning (20 tokens)": 20 * 4096,
    "IA3": 32 * (128 + 128 + 14336),   # k + v + ff per layer
    "LoRA r=8": 32 * 8 * (4096 + 4096) * 4,  # q,k,v,o projections
    "LoRA r=32": 32 * 32 * (4096 + 4096) * 4,
    "AdaLoRA avg_r=8": 32 * 8 * (4096 + 4096) * 4,  # similar to LoRA
    "VeRA r=256": 32 * (256 + 4096),    # b + d vectors per layer
    "Prefix Tuning (20 tokens)": 32 * 20 * 2 * 128,  # k,v prefix per layer
}

for name, params in configs.items():
    mb = params * 2 / 1024 / 1024  # float16
    print(f"{name:35s}: {params:>12,} params  ({mb:6.1f} MB)")

# Output:
# Prompt Tuning (20 tokens)          :       81,920 params  (  0.2 MB)
# IA3                                :      470,016 params  (  0.9 MB)
# LoRA r=8                           :   33,554,432 params  ( 64.0 MB)
# LoRA r=32                          :  134,217,728 params  (256.0 MB)
# AdaLoRA avg_r=8                    :   33,554,432 params  ( 64.0 MB)
# VeRA r=256                         :   2,703,360 params  (  5.2 MB)
# Prefix Tuning (20 tokens)          :    2,621,440 params  (  5.0 MB)

---

Common Mistakes

:::danger IA3 with the wrong learning rate IA3 scaling vectors initialize at 1.0 (identity - no change). If you use a LoRA learning rate (1e-4), the vectors barely move and the model does not adapt. Use 3e-3 to 1e-2 for IA3. This is the single most common reason IA3 "doesn't work" when someone tries it. :::

:::danger AdaLoRA without warm-up steps Setting tinit=0 means rank pruning starts at step 0, before the model has learned anything. The importance scores computed on untrained gradients are meaningless. Layers get their rank budget cut based on random noise. Always set tinit to at least 5% of total training steps - 200 steps for a 4000-step run is a good default. :::

:::warning Prompt Tuning on small models Lester et al. showed that Prompt Tuning approaches full fine-tuning quality only at model scales above 10B parameters. On 1B-3B models, the gap is 3-8% on standard benchmarks. Do not use Prompt Tuning as your only method on small models - use IA3 or LoRA instead. :::

:::warning VeRA with low rank VeRA needs a much higher rank than LoRA because the projection matrices are random and fixed - you cannot optimize them toward the task. VeRA at rank 8 is significantly worse than LoRA at rank 8. Start at rank 64-256 for VeRA. The total parameters are still much lower than LoRA because only the scaling vectors are trained, but the rank itself must be high to give the scaling vectors enough directions to work with. :::

:::warning LoftQ requires loading the model in full precision first LoftQ's alternating optimization needs access to the full-precision base weights to compute the quantization residual. If you try to run LoftQ starting from an already-quantized model (bitsandbytes 4-bit), you will get poor initialization because the residual has already been discarded. Always start LoftQ from float16 or bfloat16 weights. :::

:::danger Mixing PEFT methods naively Each PEFT method modifies the model computation graph differently. Stacking a LoRA adapter on top of a Prefix-Tuned model requires careful ordering and is not supported by default in the PEFT library. If you need multi-task composition, use LoRA's add_weighted_adapter (covered in the next lesson) rather than stacking different PEFT method types. :::

---

Interview Q&A

Q1: What is the fundamental difference between additive PEFT methods (LoRA, Prefix Tuning) and reparameterization methods (IA3)?

Additive methods introduce new parameters that are added to or concatenated with the existing computation graph. LoRA adds matrix products $BA$ to weight updates. Prefix Tuning prepends virtual key-value pairs to the attention. These increase the effective number of parameters involved in the forward pass.

Reparameterization methods (IA3) instead modify how existing activations are scaled - the computation graph topology stays the same. IA3 multiplies existing key, value, and feed-forward activations by learned scaling vectors. Nothing is added to the model; dimensions are reweighted. This is why IA3 has zero inference overhead after merging: you fold the scaling into the existing weight matrices with a single element-wise multiply during post-processing.

The practical implication: reparameterization methods are generally faster at inference and cheaper to serve, but less expressive for complex behavioral changes.

---

Q2: How does AdaLoRA's importance scoring work, and what prevents it from pruning all the rank budget to a few layers?

AdaLoRA computes an importance score for each singular value in its SVD decomposition of $\Delta W = P\Lambda Q^T$. The score combines the magnitude of the singular value and the sensitivity of the loss to that singular value:

$S_{ij} = |(\lambda_i)| \cdot \|\nabla_{\lambda_i} \mathcal{L}\|$

The score uses exponential moving averages ($\beta_1$, $\beta_2$) to smooth out noisy gradient estimates. Low-scoring singular values are masked (effectively set to zero), reducing that layer's effective rank.

To prevent collapse to one or two layers, AdaLoRA enforces a minimum rank per layer (typically 1 or 2) and uses a global budget constraint that is realized gradually over tinit to tfinal steps. The gradual schedule prevents sudden catastrophic pruning. In practice, natural regularization from the training signal also prevents extreme concentration - a model that has zeroed all rank in all but two layers loses too much representational capacity to minimize the training loss effectively.

---

Q3: A colleague proposes using Prompt Tuning for a customer support classification task on a Llama-3 8B model. What would you recommend and why?

Prompt Tuning at 8B is borderline. Lester et al.'s scaling law data suggests 8B is close to where Prompt Tuning starts to become competitive, but the task matters too. Classification tasks with clear, consistent patterns (sentiment, intent detection with well-defined intents) are favorable. Classification with subtle distinctions or domain-specific jargon is unfavorable.

My recommendation: start with IA3. It costs roughly 900K parameters (vs 82K for Prompt Tuning), which is still tiny, but gives the model access to per-layer adaptation through key and value scaling. It is much more robust than Prompt Tuning across model sizes and task types. Run a 500-step IA3 experiment and check validation accuracy. If it hits your target, ship it. If it falls short by more than 2%, switch to LoRA rank 8 and retrain. The total experimental cost is one short training run.

Avoid Prompt Tuning as the first choice for any model under 10B unless storage constraints are extreme (you need 1000+ variants per day and 80KB per adapter is genuinely a hard requirement).

---

Q4: What problem does LoftQ solve that standard QLoRA initialization does not?

Standard QLoRA initializes LoRA $A$ randomly (from a Gaussian) and $B$ to zero. The zero initialization means the adapter starts as a no-op: $\Delta W = BA = 0$ at step 0. The training then has to simultaneously learn to compensate for quantization error and learn the task signal.

Quantization error is not random - it has structure (it is the difference between the original weight and its nearest quantized representation). Random initialization ignores this structure entirely.

LoftQ finds the LoRA initialization that minimizes $\|W - Q(W) - BA\|_F$, the Frobenius norm of the residual between the original weight and the quantized weight plus the LoRA matrices. By SVD-decomposing the residual $W - Q(W)$, LoftQ sets $B$ and $A$ to explicitly cancel the quantization error at initialization. Training then starts from a position where the model already approximately recovers the original weight behavior, and only needs to learn the task signal from there.

The result is faster convergence (fewer steps to reach the same quality) and better final quality, especially at aggressive quantization levels (4-bit or lower) where quantization error is large.

---

Q5: When would you choose VeRA over LoRA for a production deployment?

VeRA's core advantage is that all adapters share the same frozen random matrices, so the marginal storage cost per additional task adapter is just two small scaling vectors (roughly 50KB for Llama-3 8B vs 64MB for LoRA rank 8). This matters in three specific scenarios:

First, continual learning systems where new task adapters are added frequently. If you fine-tune 500 task-specific variants of a base model, LoRA costs 32GB of adapter storage. VeRA costs roughly 50MB for the shared matrices (stored once) plus 25MB for 500 adapter vector sets.

Second, multi-tenant inference servers where different users have different personalization. Each user's "personal adapter" is just a pair of small scaling vectors. Hot-swapping between users costs almost nothing.

Third, federated learning across many edge devices. The shared random matrices can be distributed once. Each device then trains and transmits only its small scaling vectors, dramatically reducing communication overhead.

VeRA's weakness is that it needs a higher rank (64-512 instead of 8-64) to compensate for the fixed random projections, and convergence is typically slower than LoRA. If you are training one or a few adapters and storage is not a constraint, LoRA is almost always the better choice.

---

Q6: How does Prefix Tuning differ from adding special tokens during tokenization?

Adding special tokens during tokenization creates discrete tokens with embeddings that are looked up from the embedding table. These are constrained to the discrete vocabulary - they represent real tokens like "[SUMMARIZE]" or "[SENTIMENT]". They interact with the model only at the input layer through the embedding lookup.

Prefix Tuning creates continuous vectors that live in the same high-dimensional space as token representations but are not constrained to correspond to any real token. They can take any value in $\mathbb{R}^{d_{model}}$. More importantly, Prefix Tuning inserts these learnable vectors as keys and values at every transformer layer - not just the input. This means the attention mechanism at layer 16 can "see" a learned context signal that is specific to layer 16's processing stage, not just a fixed input token that gets progressively transformed.

Discrete special tokens have semantics constrained by their pre-training: "[SUMMARIZE]" activates the model's existing knowledge about summarization. Prefix Tuning's virtual tokens have no pre-training semantics - they are optimized purely to steer the model toward the desired behavior, which can represent concepts the vocabulary has no words for.