:::tip 🎮 Interactive Playground Visualize this concept: Try the Quantisation Explorer demo on the EngineersOfAI Playground - no code required. :::
Model Quantization for Production Inference
The LLM That Would Not Fit
The model is done. Six weeks of fine-tuning a 13B parameter LLaMA model on proprietary legal documents. Accuracy on the eval set: 87.3%. The legal team is thrilled. The infrastructure team is not.
A 13B parameter model in FP32 (4 bytes per parameter) weighs 52GB. In FP16, 26GB. A single A100 80GB can load the FP16 model - barely - but inference for a single 2000-token context takes 4.2 seconds. The product SLA is 800ms. And an A100 costs $3.50 per hour. Deploying multiple GPUs per request is economically impossible for a startup.
The ML engineer looks at the options. More powerful hardware does not exist in the needed quantity on short notice. Knowledge distillation into a smaller model would take months of retraining. The third option: quantization. Convert the weights from FP16 (16-bit floating point) to INT8 (8-bit integer) or INT4 (4-bit integer).
She runs GPTQ quantization to INT4 overnight. The quantized model weighs 7.3GB - fits on a single A10G (24GB, $0.75/hour). Single-request latency: 0.9 seconds. Not quite at SLA, but with dynamic batching and a faster tokenizer, 760ms average. The legal team cannot tell the difference between the quantized and the full-precision model. The accuracy gap is 0.8 percentage points. The cost difference is 78%.
Quantization is not a free lunch - accuracy always takes some hit. But in the right regime, the tradeoffs are so favorable that running a quantized model is the correct engineering decision for nearly every production deployment.
Why This Exists - The Memory Wall
Neural networks store parameters as floating-point numbers. The IEEE 754 standard defines FP32 as 32 bits: 1 sign bit, 8 exponent bits, 23 mantissa bits. This gives ~7 decimal digits of precision across a range of roughly to .
Most of that precision is wasted for neural network weights. A weight that is 0.4731892 and 0.4731893 will produce effectively identical outputs - the model cannot possibly have learned to distinguish them. Empirically, neural network weights cluster tightly and can be represented with far fewer bits with minimal loss of model quality.
The memory cost of FP32 weights determines everything downstream:
- How much GPU VRAM you need
- Whether a model fits on one GPU or needs tensor parallelism
- How much data you transfer per inference (bandwidth-bound operations)
- The latency of memory-bound layers (most of transformer inference is memory-bound, not compute-bound)
Quantization exploits the fact that models are not sensitive to the precision of individual weights - only their approximate value matters. By representing weights with fewer bits, you reduce memory by 2-8× and often achieve proportional latency improvements for memory-bound operations.
Historical Context
The first quantization work for neural networks appeared around 2015-2016, motivated by edge deployment (mobile devices, embedded systems) where 4GB+ of model weights were obviously impossible. Ternary Weight Networks (Zhu et al., 2016) showed that weights could be quantized to {-1, 0, +1} with surprisingly small accuracy loss.
TensorFlow Lite (2017) popularized INT8 quantization for mobile deployment, establishing the post-training quantization (PTQ) paradigm: train in FP32, quantize after training without retraining.
The LLM era created a new urgency. LLaMA-65B at FP16 is 130GB - requiring eight 80GB A100s for a single model. GPTQ (Frantar et al., 2022) showed that LLMs could be quantized to 4 bits using second-order optimization to minimize quantization error, enabling single-GPU deployment of 30B+ models. AWQ (Lin et al., 2023) improved on GPTQ by protecting the most salient weights from quantization, achieving better accuracy at the same bit-width.
GGUF (successor to GGML, 2023) introduced a flexible file format that bundles quantized weights with model metadata, enabling efficient CPU inference with mixed quantization per layer - heavier quantization for attention layers, lighter quantization for FFN layers that are more accuracy-sensitive.
The Quantization Math
Quantization maps a floating-point value to an integer using a scale factor and zero point :
And dequantization recovers an approximation:
The scale is chosen to map the range of observed values to the quantized range:
For symmetric quantization (zero point , used for weights):
The quantization error is:
The maximum error is bounded by - so a smaller scale (tighter range) reduces error. This creates the fundamental tension: if your weights have a few outliers with large magnitude, the scale must accommodate them, which coarsens the resolution for all other values.
Post-Training Quantization (PTQ)
PTQ quantizes a trained model without retraining. The weights are calibrated using a small dataset (100-1000 examples) to determine the optimal scale factors.
Weight-Only Quantization
The simplest form: quantize weights to INT8 or INT4, keep activations in FP16. Inference computes: dequantize weights to FP16 → run FP16 matrix multiply → FP16 output. Memory bandwidth drops proportionally to weight size reduction; compute stays FP16.
# weight_only_quantization.py - INT8 weight quantization
import torch
import torch.nn as nn
from typing import Tuple
def quantize_weight_int8(
weight: torch.Tensor,
per_channel: bool = True
) -> Tuple[torch.Tensor, torch.Tensor]:
"""
Quantize a weight tensor to INT8.
Args:
weight: FP32 or FP16 weight tensor, shape (out_features, in_features)
per_channel: if True, use one scale per output channel (better quality)
Returns:
weight_int8: quantized INT8 weights
scale: FP32 scale factors, shape (out_features, 1) or scalar
"""
if per_channel:
# Scale per output channel - captures per-channel weight distributions
max_abs = weight.abs().max(dim=1, keepdim=True).values
else:
max_abs = weight.abs().max()
# Symmetric quantization: range [-127, 127] (leave -128 unused to avoid asymmetry)
scale = max_abs / 127.0
scale = scale.clamp(min=1e-8) # avoid division by zero
# Quantize
weight_int8 = (weight / scale).round().clamp(-127, 127).to(torch.int8)
return weight_int8, scale.to(torch.float32)
def dequantize_weight(
weight_int8: torch.Tensor,
scale: torch.Tensor
) -> torch.Tensor:
"""Recover FP16 weight from INT8 + scale."""
return (weight_int8.to(torch.float16) * scale.to(torch.float16))
class QuantizedLinear(nn.Module):
"""Drop-in replacement for nn.Linear with INT8 weight quantization."""
def __init__(self, linear: nn.Linear, per_channel: bool = True):
super().__init__()
self.in_features = linear.in_features
self.out_features = linear.out_features
# Quantize weights at initialization time
weight_int8, scale = quantize_weight_int8(
linear.weight.data.float(), per_channel
)
self.register_buffer('weight_int8', weight_int8)
self.register_buffer('scale', scale)
if linear.bias is not None:
self.register_buffer('bias', linear.bias.data)
else:
self.bias = None
def forward(self, x: torch.Tensor) -> torch.Tensor:
# Dequantize weights for each forward pass
weight_fp16 = dequantize_weight(self.weight_int8, self.scale)
return nn.functional.linear(x, weight_fp16, self.bias)
def quantize_model_int8(model: nn.Module, per_channel: bool = True) -> nn.Module:
"""Replace all Linear layers with QuantizedLinear."""
for name, module in model.named_children():
if isinstance(module, nn.Linear):
setattr(model, name, QuantizedLinear(module, per_channel))
else:
quantize_model_int8(module, per_channel) # recurse
return model
Activation Quantization with Calibration
For full INT8 (weights and activations), you need calibration data to determine activation ranges:
# calibration.py - collect activation ranges for PTQ
import torch
import torch.nn as nn
from collections import defaultdict
from typing import Dict, List
class ActivationObserver:
"""Collects statistics for a single layer's activations during calibration."""
def __init__(self, percentile: float = 99.9):
self.percentile = percentile
self.min_vals: List[float] = []
self.max_vals: List[float] = []
def observe(self, tensor: torch.Tensor):
"""Record min/max of this batch's activations."""
p_low = (100 - self.percentile) / 2
p_high = 100 - p_low
self.min_vals.append(torch.quantile(tensor.float(), p_low / 100).item())
self.max_vals.append(torch.quantile(tensor.float(), p_high / 100).item())
def get_scale_zero_point(self) -> Tuple[float, int]:
"""Compute INT8 scale and zero_point from observed range."""
x_min = min(self.min_vals)
x_max = max(self.max_vals)
scale = (x_max - x_min) / 255.0
zero_point = round(-x_min / scale) - 128
zero_point = max(-128, min(127, zero_point))
return scale, zero_point
def calibrate_model(
model: nn.Module,
calibration_loader: torch.utils.data.DataLoader,
num_batches: int = 100
) -> Dict[str, ActivationObserver]:
"""
Run calibration data through the model, collect activation stats.
Returns dict mapping layer_name -> observer with computed ranges.
"""
observers = defaultdict(lambda: ActivationObserver())
hooks = []
def make_hook(name):
def hook(module, input, output):
observers[name].observe(output.detach())
return hook
# Register hooks on all linear layers
for name, module in model.named_modules():
if isinstance(module, nn.Linear):
hooks.append(module.register_forward_hook(make_hook(name)))
model.eval()
with torch.no_grad():
for i, batch in enumerate(calibration_loader):
if i >= num_batches:
break
inputs = batch[0].to(next(model.parameters()).device)
model(inputs)
if i % 10 == 0:
print(f"Calibration batch {i}/{num_batches}")
# Remove hooks
for hook in hooks:
hook.remove()
return observers
GPTQ: Quantization for Large Language Models
GPTQ (2022) uses second-order information (the Hessian of the loss with respect to weights) to choose which weight bits to sacrifice. The insight: some weights affect the output much more than others - protect high-importance weights, sacrifice low-importance ones.
The algorithm processes layers sequentially, quantizing column by column and compensating for the error introduced by each quantized column:
# gptq_example.py - using the auto-gptq library
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer
import torch
model_name = "meta-llama/Llama-2-13b-hf"
# Configure GPTQ quantization
quantize_config = BaseQuantizeConfig(
bits=4, # 4-bit quantization
group_size=128, # quantize in groups of 128 weights
# (smaller groups = better quality, more overhead)
desc_act=True, # reorder activations for better quantization
damp_percent=0.01, # Hessian damping for numerical stability
sym=True, # symmetric quantization
)
# Load model in FP16
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoGPTQForCausalLM.from_pretrained(
model_name,
quantize_config=quantize_config,
torch_dtype=torch.float16,
)
# Calibration data - 128 samples is typically enough
calibration_data = [
tokenizer(text, return_tensors="pt").input_ids
for text in load_calibration_texts(num_samples=128)
]
# Run GPTQ quantization (takes 30-120 minutes for 13B model)
model.quantize(
calibration_data,
batch_size=4,
use_triton=True, # faster with Triton kernels
)
# Save quantized model
model.save_quantized("llama-13b-gptq-4bit")
# Load and run inference
quantized_model = AutoGPTQForCausalLM.from_quantized(
"llama-13b-gptq-4bit",
use_safetensors=True,
device="cuda:0",
inject_fused_attention=True, # fused kernels for speed
)
output = quantized_model.generate(
tokenizer("Explain quantum computing:", return_tensors="pt").input_ids.cuda(),
max_new_tokens=200
)
print(tokenizer.decode(output[0]))
AWQ: Activation-Aware Weight Quantization
AWQ (2023) improves over GPTQ with a key insight: not all weights are equally important. Weights that correspond to activation channels with high magnitude are more sensitive to quantization error - protecting those channels dramatically improves quantized model quality.
AWQ finds the 1% of weight channels that are most activation-sensitive and scales them up before quantization (and scales activations down correspondingly), preserving their precision without changing the actual quantization bit-width.
# awq_example.py - using the awq library
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_name = "meta-llama/Llama-2-13b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoAWQForCausalLM.from_pretrained(model_name, trust_remote_code=True)
quant_config = {
"zero_point": True, # asymmetric quantization
"q_group_size": 128, # group size for per-group quantization
"w_bit": 4, # 4-bit weights
"version": "GEMM", # use GEMM kernel (vs GEMV for small batch)
}
# AWQ quantization - typically faster than GPTQ (no per-column Hessian)
model.quantize(
tokenizer,
quant_config=quant_config,
calib_data="pileval" # uses a built-in calibration dataset
)
model.save_quantized("llama-13b-awq-4bit")
tokenizer.save_pretrained("llama-13b-awq-4bit")
AWQ vs GPTQ comparison:
| Aspect | GPTQ | AWQ |
|---|---|---|
| Quantization time (13B) | 60-120 min | 15-30 min |
| Accuracy at 4-bit | Good | Slightly better |
| Inference kernel | ExllamaV2, AutoGPTQ | Faster GEMM kernels |
| Group size flexibility | 32-128 | 64-128 |
| CPU offload support | Limited | Yes (via llama.cpp) |
GGUF Format for Flexible Deployment
GGUF (GPT-Generated Unified Format) is the file format used by llama.cpp. It supports mixed quantization - different layers can use different bit-widths - and enables efficient CPU inference with partial GPU offloading.
# using llama-cpp-python with GGUF models
from llama_cpp import Llama
# Load Q4_K_M quantized model (4-bit, K-quantization, medium quality)
llm = Llama(
model_path="llama-2-13b.Q4_K_M.gguf",
n_gpu_layers=35, # offload 35 layers to GPU (rest on CPU)
n_ctx=4096, # context window
n_batch=512, # batch size for prompt processing
verbose=False,
)
# Generate
output = llm(
"Explain transformer attention in one paragraph:",
max_tokens=200,
temperature=0.7,
echo=False
)
print(output['choices'][0]['text'])
# GGUF quantization types and their tradeoffs:
GGUF_TYPES = {
"Q2_K": {"bits": 2.63, "size_gb": 3.4, "perplexity_loss": "~15%"},
"Q3_K_M": {"bits": 3.35, "size_gb": 4.3, "perplexity_loss": "~5%"},
"Q4_K_M": {"bits": 4.45, "size_gb": 5.7, "perplexity_loss": "~1.5%"},
"Q5_K_M": {"bits": 5.45, "size_gb": 6.9, "perplexity_loss": "~0.5%"},
"Q6_K": {"bits": 6.56, "size_gb": 8.3, "perplexity_loss": "~0.1%"},
"Q8_0": {"bits": 8.50, "size_gb": 10.7, "perplexity_loss": "~0.05%"},
# For reference
"F16": {"bits": 16.0, "size_gb": 26.0, "perplexity_loss": "0%"},
}
# Q4_K_M is the typical production sweet spot for LLaMA-13B:
# 5.7GB, fits on a single 8GB VRAM GPU, acceptable 1.5% perplexity loss
Quantization-Aware Training (QAT)
QAT simulates quantization noise during training, allowing the model to adapt its weights to minimize quantization error. It typically achieves better accuracy than PTQ at the cost of retraining time.
# qat_example.py - quantization-aware training with PyTorch
import torch
import torch.nn as nn
from torch.quantization import (
get_default_qat_qconfig,
prepare_qat,
convert
)
class SmallClassifier(nn.Module):
def __init__(self, input_dim: int, num_classes: int):
super().__init__()
self.quant = torch.quantization.QuantStub() # quantize inputs
self.fc1 = nn.Linear(input_dim, 256)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(256, num_classes)
self.dequant = torch.quantization.DeQuantStub() # dequantize outputs
def forward(self, x):
x = self.quant(x)
x = self.fc1(x)
x = self.relu(x)
x = self.fc2(x)
x = self.dequant(x)
return x
model = SmallClassifier(input_dim=512, num_classes=10)
# Step 1: Configure QAT - fake quantization nodes inserted in graph
model.qconfig = get_default_qat_qconfig('fbgemm') # x86 CPU
model_prepared = prepare_qat(model.train())
# Step 2: Train with fake quantization active
# Model learns to be robust to INT8 noise during this training phase
optimizer = torch.optim.Adam(model_prepared.parameters(), lr=1e-4)
for epoch in range(10):
for batch_x, batch_y in train_loader:
logits = model_prepared(batch_x)
loss = nn.CrossEntropyLoss()(logits, batch_y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Step 3: Convert to actual quantized model (INT8 weights + activations)
model_prepared.eval()
model_quantized = convert(model_prepared)
# Save
torch.save(model_quantized.state_dict(), "model_qat_int8.pt")
# The quantized model uses integer arithmetic on CPU
# Typical speedup: 2-4× on x86 CPUs with FBGEMM or QNNPACK backend
Accuracy vs Latency Tradeoffs
For LLMs specifically (LLaMA-13B on Wikitext-2 perplexity - lower is better):
| Format | Perplexity | Size (GB) | Tokens/sec (A10G) |
|---|---|---|---|
| FP16 | 5.47 | 26.0 | 28 |
| GPTQ INT8 | 5.51 | 13.0 | 51 |
| GPTQ INT4 | 5.63 | 7.3 | 89 |
| AWQ INT4 | 5.58 | 7.3 | 96 |
| GGUF Q4_K_M | 5.69 | 5.7 | 72 (CPU+GPU) |
The INT4 models achieve 3-4× higher throughput at 2.2-4% perplexity increase - a favorable tradeoff for most production use cases.
Calibration Dataset Selection
PTQ accuracy depends critically on the calibration dataset. The calibration data should match your production distribution:
# calibration_data.py - building a good calibration dataset
from datasets import load_dataset
import random
def build_calibration_dataset(
target_domain: str,
num_samples: int = 512,
seq_len: int = 512,
tokenizer = None
) -> list:
"""
Build calibration dataset for model quantization.
Rule: use data similar to your production inputs.
"""
if target_domain == "general_text":
# Pileval is a common calibration dataset
dataset = load_dataset("mit-han-lab/pile-val-backup", split="validation")
texts = [item["text"] for item in dataset.select(range(num_samples * 5))]
elif target_domain == "code":
dataset = load_dataset("codeparrot/github-code", split="train",
streaming=True)
texts = [item["code"] for _, item in zip(range(num_samples * 5), dataset)]
elif target_domain == "legal":
# Use representative samples from your actual data
# CRITICAL: do not use test set - calibration data should be from train/val
texts = load_your_legal_corpus(num_samples * 5)
# Tokenize and truncate/pad to fixed length
calibration_tokens = []
for text in texts:
tokens = tokenizer(text, truncation=True, max_length=seq_len,
return_tensors="pt").input_ids
if tokens.shape[1] >= seq_len // 2: # skip very short texts
calibration_tokens.append(tokens)
if len(calibration_tokens) >= num_samples:
break
return calibration_tokens
# Key rules for calibration data:
# 1. At least 128 samples - more doesn't help much beyond ~512
# 2. Use production-representative data, not the training set
# 3. Cover edge cases: long sequences, rare tokens, domain-specific vocabulary
# 4. Do NOT use the test set - calibration can overfit to test distribution
Production Engineering Notes
Choosing Between PTQ and QAT
Use PTQ (post-training quantization) when:
- You cannot afford retraining time (most common case)
- Model is larger than ~1B parameters (QAT is expensive to train)
- Accuracy drop from PTQ is acceptable for your task (check with eval)
Use QAT (quantization-aware training) when:
- You have a smaller model (under 100M parameters)
- PTQ accuracy drop is unacceptable
- You are targeting edge deployment where INT8 compute savings matter most
- You have access to training infrastructure and training data
Verifying Quantization Quality
Always benchmark quantized models on your actual task metrics, not just perplexity:
# quantization_eval.py - compare full vs quantized model
import torch
from typing import Dict
def evaluate_models(
full_model: torch.nn.Module,
quantized_model: torch.nn.Module,
eval_loader: torch.utils.data.DataLoader,
device: str = "cuda"
) -> Dict[str, Dict]:
results = {}
for model_name, model in [("full_fp16", full_model), ("quantized_int4", quantized_model)]:
model.eval()
correct = 0
total = 0
total_latency_ms = 0.0
with torch.no_grad():
for batch_x, batch_y in eval_loader:
batch_x = batch_x.to(device)
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
output = model(batch_x)
end.record()
torch.cuda.synchronize()
latency_ms = start.elapsed_time(end)
total_latency_ms += latency_ms
preds = output.argmax(dim=1).cpu()
correct += (preds == batch_y).sum().item()
total += len(batch_y)
results[model_name] = {
"accuracy": correct / total,
"avg_latency_ms": total_latency_ms / len(eval_loader),
}
print(f"{model_name}: accuracy={results[model_name]['accuracy']:.4f}, "
f"latency={results[model_name]['avg_latency_ms']:.1f}ms")
accuracy_drop = results["full_fp16"]["accuracy"] - results["quantized_int4"]["accuracy"]
speedup = results["full_fp16"]["avg_latency_ms"] / results["quantized_int4"]["avg_latency_ms"]
print(f"\nAccuracy drop: {accuracy_drop:.4f} ({accuracy_drop*100:.2f}%)")
print(f"Speedup: {speedup:.1f}×")
return results
Common Mistakes
:::danger Quantizing Without Calibration Data Running PTQ without calibration (just using the theoretical min/max of INT8 range) produces poor results - scale factors are wildly incorrect for the actual weight/activation distributions. Always run at least 128 representative samples through the model before quantization. The calibration step takes 5-15 minutes and dramatically improves accuracy. :::
:::danger Using Training Data for Calibration Calibration data should come from your validation set, not training set. Using training data biases the scale factors toward the training distribution, which may not match production inputs. If your training data is under NDA, use a domain-representative public dataset as a proxy. :::
:::warning Quantizing Normalization Layers Batch normalization and layer normalization layers are numerically sensitive and should typically stay in FP32 or FP16 even when the rest of the model is INT8. Most quantization frameworks skip these layers automatically. If implementing custom quantization, explicitly exclude them. :::
:::warning Outlier Activation Channels Break INT8 Some models (especially LLMs like OPT) have systematic activation outliers - channels where values are 10-100× larger than typical channels. These outliers force the scale factor wide, coarsening the resolution for all normal-range channels. SmoothQuant (2022) migrates this difficulty from activations to weights (which are easier to quantize), solving the INT8 LLM problem without accuracy loss. Use SmoothQuant before INT8 quantization for LLMs. :::
Interview Q&A
Q1: What is the difference between symmetric and asymmetric quantization, and when would you use each?
A: Symmetric quantization uses (zero-point at zero) and maps the range to . Asymmetric quantization uses a non-zero zero-point to shift the range, mapping to . For weights (which are roughly symmetric around zero due to regularization), symmetric quantization works well and is computationally simpler - integer matrix multiplies do not need to handle zero-point offsets. For activations (which are often positive after ReLU, range ), asymmetric is better because symmetric would waste half the quantization range representing negative values that never occur. Most production frameworks use symmetric for weights and asymmetric for activations.
Q2: How does GPTQ achieve better quantization quality than naive round-to-nearest?
A: GPTQ uses second-order information - the Hessian of the layer's loss with respect to weights - to minimize the quantization error in a principled way. The key insight is that weights are not equally important: some weights, when perturbed, cause large changes in output; others cause tiny changes. GPTQ quantizes one column of the weight matrix at a time, then compensates for the introduced error in the remaining unquantized columns using the inverse Hessian. This is similar to optimal brain compression (OBC) but made tractable for billion-parameter models through batching and lazy update approximations. The result is that the most impactful weights get the best precision assignment, while less important weights absorb more quantization error. In practice, GPTQ INT4 achieves similar perplexity to naive INT8 round-to-nearest.
Q3: Explain the speedup mechanism for quantized inference. Why does INT4 not give 8× speedup even though it is 8× fewer bits?
A: The speedup from quantization comes from two sources: memory bandwidth reduction and compute acceleration. A) Memory bandwidth: transformer inference is largely memory-bandwidth-bound - the time to load weights from GPU DRAM dominates computation time. INT4 weights are 8× smaller, so they load in 1/8th the time. This directly translates to ~8× speedup for memory-bound operations, which is why weight-only INT4 quantization approaches 8× throughput improvement for LLMs. B) Compute: if you actually perform INT4 arithmetic (not just dequantize to FP16), specialized INT4 kernels can process 2× more elements per clock compared to INT8, and 8× compared to FP32. But this requires hardware support (H100's FP8/INT4 tensor cores) and efficient kernel implementations. In practice, many INT4 implementations dequantize to FP16 at compute time, capturing only the memory bandwidth benefit (~4-6× for typical mixed workloads). True compute-level INT4 requires hardware-specific kernels (CUTLASS, cuBLAS INT4 extensions) and is only available on newer GPU architectures.
Q4: How do you decide which quantization approach to use for a production LLM deployment?
A: The decision tree: First, check if FP16 fits on available hardware - if yes and latency is acceptable, stay at FP16 (zero accuracy cost). If not, try GPTQ or AWQ INT4: GPTQ has more calibration options (group size, dampening) for tweaking accuracy; AWQ is faster to quantize and often slightly better quality. Both give similar final accuracy. If you need CPU inference or mixed CPU/GPU deployment, GGUF Q4_K_M via llama.cpp is the practical choice - it has optimized kernels for AVX2/AVX-512 CPU inference and supports partial GPU offloading. For latency-critical paths, measure: INT4 with optimized GEMM kernels (ExllamaV2, AWQ GEMM) typically gives the best tokens/second. Always validate on your actual task - perplexity is a proxy, not a guarantee. Some task-specific accuracy (classification, structured output) degrades more than perplexity suggests.
Q5: What is the accuracy-latency Pareto frontier for LLM quantization, and how do you navigate it?
A: The Pareto frontier is the set of configurations where you cannot improve accuracy without increasing latency (or memory), and cannot reduce latency without sacrificing accuracy. For LLM quantization, the frontier looks approximately like: FP16 (best accuracy, highest latency/memory), INT8 (very small accuracy drop, ~2× latency improvement), INT4 (small accuracy drop, ~4-6× latency improvement), INT2/ternary (significant accuracy drop, ~8-12× improvement). Most production systems operate at INT4 because it sits at the most favorable tradeoff - the accuracy drop (1-3%) is acceptable for most tasks while the latency and cost improvement is substantial. The navigation strategy: start by quantifying the acceptable accuracy drop for your task (measure on held-out eval set, talk to product about what degradation is user-noticeable), then find the lowest-bit quantization that stays within that threshold. For revenue-critical tasks (search ranking, fraud detection), the threshold may be 0.1%; for general assistants, 2-3% is typically acceptable.