LLM Quantization and Model Compression: The Complete Guide

Introduction

Large language models have grown from 175 billion parameters (GPT-3) to over a trillion parameters in 2026. At FP32 precision, a single 70B-parameter model requires 280 GB of GPU memory — beyond what any single consumer or server GPU can hold. This is the fundamental tension driving the field of model compression: capability grows with scale, but deployment demands efficiency.

Model compression bridges this gap. By reducing the precision of model weights (quantization), removing redundant parameters (pruning), or transferring knowledge to smaller architectures (distillation), compression makes it possible to run state-of-the-art models on consumer hardware, achieve sub-100ms latency in production, and reduce inference costs by 4-8x. In 2026, no production LLM deployment skips compression — the question is which technique to use and how aggressively to apply it [1].

This guide covers the full landscape of LLM quantization and compression: the mathematical foundations of quantization, the major techniques (GPTQ, AWQ, GGUF, PTQ, QAT), weight pruning and sparse models, knowledge distillation, and the practical trade-offs between compression ratio, accuracy, and latency. We also cover production deployment with tools like vLLM, llama.cpp, and TensorRT-LLM, and provide guidance for choosing the right approach based on your deployment constraints.

Quantization Fundamentals

Numerical Precision in Deep Learning

Neural network weights are stored as floating-point numbers. The precision of these numbers determines both model quality and memory footprint. Full precision (FP32) uses 32 bits per weight — 1 sign bit, 8 exponent bits, and 23 mantissa bits — providing roughly 7 decimal digits of precision. Half precision (FP16) uses 16 bits (5 exponent, 10 mantissa), while the more recent BF16 uses 16 bits with 8 exponent and 7 mantissa bits, matching FP32's dynamic range with reduced precision [2].

Integer quantization maps floating-point weights to integer representations — typically 8-bit (INT8), 4-bit (INT4), or even 2-bit and 1-bit. The mapping can be symmetric (weights map to a range centered at zero) or asymmetric (weights map to a range shifted by a zero-point). Symmetric quantization is simpler and works well for weights that are roughly symmetric around zero — which is typical in trained neural networks. Asymmetric quantization preserves more dynamic range for skewed distributions and is commonly used for activations.

Calibration Datasets

Quantization is not purely a mathematical operation — it requires calibration data to determine the optimal scaling factors. The calibration dataset should be representative of the model's expected input distribution. For language models, this typically means 128-512 samples drawn from the model's training corpus or a domain-specific dataset. The calibration process runs a forward pass through the model on the calibration data, observes the activation distributions, and computes scaling factors that minimize the information loss from quantization [3].

The choice of calibration data significantly impacts quantization quality. Using generic Wikipedia text to calibrate a code model will produce worse results than using a sample of code from GitHub. Teams deploying compressed models in production should invest in curation of representative calibration datasets — the quality difference can be 1-3% in downstream task accuracy.

Weight Clipping and Outlier Handling

A critical challenge in LLM quantization is the presence of outlier features — weights or activations with magnitudes significantly larger than the typical distribution. Modern LLMs consistently show approximately 5-10 outlier channels per transformer layer where values are 10-100x larger than the median. Standard quantization schemes allocate most representational capacity to normal values, and outliers either get clipped (losing information) or overflow.

Techniques like LLM.int8() decompose matrix multiplication into two parts: normal values processed in INT8 and outlier values processed in FP16 [4]. SmoothQuant addresses the problem by smoothing the quantization difficulty between weights and activations, shifting the quantization burden from activations (which have more severe outliers) to weights (which are easier to quantize). These techniques make INT8 quantization practical for models up to 175B parameters without accuracy degradation.

Quantization Techniques

Post-Training Quantization (PTQ)

PTQ is the simplest quantization approach: take a trained model, run calibration, and produce quantized weights without any retraining. Dynamic PTQ computes scaling factors at runtime based on activation statistics. Static PTQ computes scaling factors offline using calibration data. Static PTQ is preferred for production because it eliminates runtime overhead and produces consistent results.

PTQ to INT8 typically causes less than 1% accuracy degradation for models with 7B+ parameters — the larger the model, the more robust it is to quantization. PTQ to INT4, however, can cause 2-5% degradation and requires more sophisticated techniques. PTQ is the fastest path to a compressed model and is appropriate when you need results quickly and can accept modest accuracy loss.

Quantization-Aware Training (QAT)

QAT incorporates quantization effects during training. The model is trained with fake-quantization operations that simulate the behavior of quantized inference. During forward passes, weights are quantized and dequantized, forcing the model to learn representations that are robust to quantization noise. QAT typically recovers 1-3% of the accuracy lost during PTQ, particularly for smaller models and aggressive quantization levels [5].

The main cost of QAT is computational: it requires a full training pass over the model with modified forward passes. For a 7B model, this costs approximately $5,000-15,000 in compute. Most teams reserve QAT for scenarios where accuracy requirements are strict or when deploying to edge devices where aggressive quantization (INT4 or below) is necessary.

GPTQ — Post-Training Quantization for Generative Models

GPTQ, introduced by Frantar et al. in 2023, is a one-shot weight quantization method that achieves high-quality INT4 quantization for generative language models [6]. GPTQ works by solving a layer-wise quantization problem: for each layer, it quantizes weights one column at a time, adjusting remaining weights to compensate for the quantization error of each column. This compensation mechanism is what distinguishes GPTQ from naive quantization — it distributes the error across multiple weights rather than concentrating it in the quantized weight alone.

GPTQ with group size 128 achieves 4-bit quantization with less than 1% perplexity degradation on most models. It has become the default quantization method for GPU-based inference, supported natively in vLLM, AutoGPTQ, and Hugging Face Transformers. The quantization process takes 1-4 hours for a 70B model on a single A100, making it practical for most development workflows.

AWQ — Activation-Aware Weight Quantization

AWQ, introduced by Lin et al. in 2024, improves on GPTQ by observing that not all weights contribute equally to model quality — a small fraction of weights (salient channels) have a disproportionate impact on output quality [7]. AWQ identifies these salient channels by examining activation magnitudes and preserves them at higher precision (FP16) while quantizing the remaining weights more aggressively. This selective precision approach achieves better accuracy than uniform quantization at the same average bit width.

In practice, AWQ outperforms GPTQ by 0.5-1% on standard benchmarks at INT4, with the gap widening at more aggressive quantization levels. AWQ is supported in vLLM v1 and TensorRT-LLM, and is the recommended quantization method for latency-critical production deployments where every fraction of a percent of accuracy matters.

GGML and GGUF — CPU-Optimized Quantization

GGML and its successor GGUF are quantization formats designed for CPU inference by the llama.cpp project. Unlike GPTQ and AWQ, which target GPU execution, GGUF is optimized for running on consumer hardware with no dedicated GPU. GGUF supports a wide range of quantization levels from Q2_K (2-bit) through Q8_0 (8-bit), with different schemes for different weight types (attention weights, feed-forward weights, embeddings).

The trade-off is clear: GGUF models run on any device with a CPU (including laptops, phones, and servers without GPUs), but inference is typically 5-20x slower than GPU-based inference. For interactive applications, GGUF is practical up to 13B models on modern hardware. For batch processing or server deployment, GPU-based quantization (GPTQ/AWQ) is strongly preferred. The GGUF ecosystem, including llama.cpp and Ollama, has made local LLM inference accessible to millions of users and is the primary driver of open-source LLM adoption on consumer hardware [8].

Weight Compression Beyond Quantization

Pruning

Pruning removes weights that contribute minimally to model output. Unstructured pruning zeros out individual weights based on magnitude, creating sparse weight matrices that can be stored efficiently (using sparse matrix formats) but require specialized hardware or software for actual speedup. Structured pruning removes entire neurons, attention heads, or layers, producing dense smaller matrices that provide immediate speedup on standard hardware [9].

Sparse GPT — introduced by OpenAI and others — demonstrates that post-training magnitude pruning can remove 30-50% of weights from large models with minimal accuracy degradation. The lottery ticket hypothesis suggests that dense models contain sparse subnetworks that can match the original accuracy. In practice, combining 2:4 structured sparsity (where at least 2 of every 4 weights are zero) with NVIDIA Ampere GPU hardware support enables 2x throughput improvement without accuracy loss.

Mixture-of-Experts Routing Compression

MoE models like Mixtral 8x7B and GPT-4 already use sparsity at the architectural level by routing each input to a subset of experts. Compression techniques for MoE models focus on reducing expert size (quantizing each expert), reducing the number of active experts (routing fewer experts per token), or merging experts via distillation. The observation that many experts learn similar functions opens the door to expert merging, which can reduce the effective model size by 20-30% with minimal quality loss.

Knowledge Distillation

Knowledge distillation compresses a large teacher model into a smaller student model by training the student to match the teacher's outputs. The standard approach uses logit matching — the student learns from both the ground-truth labels and the teacher's soft probabilities, which encode rich information about class relationships and relative confidences [10].

Feature-level distillation extends this by matching intermediate representations: the student learns to produce similar hidden states and attention patterns to the teacher. This is particularly effective for transformer models, where the student can be 2-4x smaller while retaining 95-98% of the teacher's performance on downstream tasks. DeepSeek R1 and Phi-3 are notable examples of successful distillation — Phi-3 compressed GPT-4's capabilities into a 3.8B parameter model that runs on phones.

Distillation has a critical advantage over quantization: it produces a permanently smaller model that requires no special runtime support. The cost is the training process itself — distillation typically costs 30-50% of the original training cost. For most teams, distillation is a strategic investment (compress once, deploy everywhere) while quantization is a tactical decision (compress each model version as needed).

Practical Impact of Compression

The memory reduction from quantization is straightforward: FP32 to INT8 is a 4x reduction, FP32 to INT4 is an 8x reduction. For a 70B model, this means reducing memory requirements from 280 GB (FP32) to 140 GB (FP16), 70 GB (INT8), or 35 GB (INT4). An RTX 4090 with 24 GB VRAM can run a 70B model at INT4 with acceptable throughput — a deployment scenario that was impossible two years ago.

Latency improvements from quantization come from two sources: reduced memory bandwidth (loading fewer bytes per weight) and, on GPUs with INT4/INT8 tensor core support, faster computation. On NVIDIA H100 GPUs, INT8 tensor cores provide 2x the throughput of FP16, and INT4 provides 4x. In practice, end-to-end inference speedups are 2-4x for INT8 and 3-6x for INT4, depending on the model, quantization method, and batch size.

Accuracy degradation varies strongly with model size. Larger models are more robust to quantization: a 70B model at INT4 typically loses 0.5-1% on MMLU accuracy, while a 7B model may lose 2-4%. Domain-specific fine-tuned models are more sensitive than base models — a medical or legal model fine-tuned on narrow distributions may lose 5-8% under INT4 quantization. This asymmetry is critical: compress your base model, measure the impact on your specific downstream tasks, and adjust quantization level accordingly.

Production Deployment

vLLM with Quantization

vLLM v1, the most widely used LLM serving framework as of 2026, provides first-class support for GPTQ, AWQ, and FP8 quantization. AWQ-quantized models running on vLLM achieve the best combination of throughput and accuracy for GPU-based deployment. vLLM's PagedAttention algorithm interacts favorably with quantized models because the KV cache (which dominates memory for long-context workloads) can also be quantized — typically to FP8 without accuracy loss [11].

Production configuration for vLLM with quantization requires tuning the quantization parameters (group size, pack size) against your latency and accuracy budgets. The default group size of 128 balances quality and efficiency, but reducing to group size 32 can recover 0.3-0.5% accuracy at the cost of 10-15% throughput.

llama.cpp for Local Inference

llama.cpp provides the most comprehensive quantization pipeline for local and edge deployment. Its quantization tool supports over 20 quantization types across the GGUF format, from Q2_K (extreme compression for small models) to Q8_0 (minimal quality loss). For edge deployment on phones or laptops, Q4_K_M (4-bit mixed precision) is the recommended starting point — it achieves 95-97% of original accuracy with 4x compression [8].

llama.cpp's server mode provides an OpenAI-compatible API, making it straightforward to swap between local and cloud inference without application changes. This is the architecture used by Ollama, LM Studio, and GPT4All.

TensorRT-LLM

NVIDIA's TensorRT-LLM provides the highest throughput for production GPU deployments by combining quantization with advanced graph optimization, kernel fusion, and in-flight batching. It supports INT4 AWQ, INT8 smooth quantization, and FP8 — the latter being particularly efficient on H100 GPUs where FP8 tensor cores deliver peak throughput [12].

TensorRT-LLM's quantization pipeline requires converting models through TensorRT's format, which adds engineering overhead compared to vLLM's native quantization support. For teams already invested in NVIDIA infrastructure, TensorRT-LLM typically provides 20-40% higher throughput than vLLM for quantized models, making the integration cost worthwhile for high-volume deployments.

Evaluating Compressed Models

Evaluation of compressed models must go beyond perplexity, which correlates only weakly with downstream task performance at aggressive quantization levels. Perplexity degradation of less than 1 point typically corresponds to no measurable task degradation. Degradation of 1-3 points requires careful evaluation on your target tasks. Degradation beyond 3 points usually indicates unacceptable quality loss.

Task-specific evaluation is essential. A compressed model may retain general knowledge while losing performance on narrow, high-stakes tasks — the opposite of what perplexity suggests. For RAG systems, compression may impact the model's ability to follow complex retrieval instructions. For chat systems, compression may flatten the model's persona adherence. We recommend maintaining a task-specific eval suite that covers your key use cases and running it against every compressed model variant before production deployment.

For teams running quantized models in production, see our guide on LLM evaluation in production for a comprehensive framework for measuring quality degradation and regression detection.

Choosing the Right Technique

The choice of compression technique depends on your deployment constraints. For latency-critical applications on modern GPUs, AWQ at INT4 with vLLM provides the best accuracy-throughput trade-off. For CPU or edge deployment, GGUF at Q4_K_M with llama.cpp is the standard. For maximum throughput on NVIDIA hardware, TensorRT-LLM with FP8 or INT4 AWQ is the production-grade choice.

For accuracy-sensitive applications (medical, legal, financial), consider FP16 or INT8 quantization only, or use QAT to recover accuracy lost at higher compression levels. For cost-sensitive batch inference, INT4 quantization provides the best cost-per-token ratio — the accuracy loss is acceptable for most use cases, and the throughput improvement directly reduces serving costs.

If you are deploying on edge devices, see our guide on edge AI and on-device ML for additional considerations around power consumption, thermal constraints, and model update strategies. For teams fine-tuning compressed models, see our fine-tuning guide for quantization-aware fine-tuning techniques.

Finally, consider whether you need open-source models at all. See our comparison of open-source vs closed-source LLMs for guidance on when proprietary APIs (which handle compression transparently) may be the more practical choice.

Tools Overview

• llama.cpp / Ollama: CPU-focused quantization pipeline (GGUF), supports over 20 quantization types, ideal for local and edge deployment.
• AutoGPTQ: Easy-to-use GPTQ quantization for Hugging Face models, supports group size and desc_act parameters.
• bitsandbytes: Quantization library for Hugging Face Transformers, supports 4-bit and 8-bit quantization with NF4 and FP4 data types.
• AutoAWQ: AWQ quantization implementation with vLLM native support, best choice for production GPU deployment.
• TensorRT-LLM: NVIDIA's optimization and quantization toolkit for high-throughput production deployment.
• vLLM v1: Production inference server with native support for GPTQ, AWQ, and FP8 quantization.
• Neural Magic (DeepSparse): CPU-based sparsity and quantization engine leveraging 2:4 structured sparsity and INT8 quantization.