LLM Inference Cost Optimization: Strategies to Reduce AI Infrastructure Spending

Introduction

Inference cost is the dominant operational expense for AI-powered products in 2026. While training a model is a one-time capital investment, inference is a recurring operational cost that scales linearly with usage. For a mid-sized deployment serving 100 million tokens per day, inference costs can exceed $200,000 per year on frontier API pricing — and this figure grows as models become larger and usage expands [1].

The good news is that inference costs are highly optimizable. A combination of quantization, batching strategy, caching, and model selection can reduce cost-per-token by 4-20x without significantly degrading output quality. This guide breaks down where inference costs come from and provides a structured playbook for reducing them, with specific techniques, benchmarks, and decision frameworks.

How Inference Costs Break Down

Understanding where inference costs come from is essential for optimizing them. The cost of a single inference call has three primary components:

Compute (GPU cycles)

Each token generated requires a forward pass through the entire model. For a 70B-parameter model at FP16 precision, a single forward pass requires approximately 140 billion floating-point operations (FLOPs) for the attention computation plus 560 billion FLOPs for the feed-forward layers — roughly 700 billion FLOPs per token. On an H100 GPU rated at 989 TFLOPS for FP16, the theoretical maximum throughput is about 1,400 tokens per second. In practice, achieving even 30-50% of theoretical peak is considered excellent due to memory bandwidth bottlenecks [2].

Memory (VRAM capacity)

The model weights themselves occupy the largest share of GPU memory. A 70B model at FP16 requires 140 GB — filling two H100s (80 GB each). The KV cache for a single conversation of 4K tokens adds roughly 2 GB. With concurrent requests, KV cache memory can exceed model memory: handling 100 concurrent 4K-token conversations requires 200 GB of KV cache memory alongside the 140 GB of model weights. This is why KV cache optimization is one of the most impactful cost-reduction strategies [3].

Bandwidth (data movement)

For autoregressive generation, memory bandwidth is the binding constraint — not compute. Generating each token requires loading the entire model weights from GPU memory into the compute units. On an H100 with 3.35 TB/s memory bandwidth, loading 140 GB of weights takes approximately 42 microseconds. This limits single-request throughput to roughly 24 tokens per second per GPU (1,000,000 microseconds / 42 microseconds per token) before accounting for attention computation time. This memory-bound property is why quantization (which reduces the bytes per weight) delivers near-linear throughput improvements [4].

Batch vs Real-Time Inference Trade-Offs

The choice between batch and real-time inference is one of the most consequential cost decisions in AI infrastructure. Batch processing queues requests and processes them together, maximizing GPU utilization. Real-time processing serves each request individually, minimizing latency but wasting GPU cycles on idle time.

At batch size 1, GPU utilization for LLM inference typically ranges from 5-15%. Each request loads the full model weights, generates the output, and then sits idle while waiting for the next request. At batch size 64, utilization reaches 80-90% because the same model weights serve 64 simultaneous requests — the cost of loading weights is amortized across all requests in the batch [5].

The throughput scaling is nearly linear: batch size 16 achieves roughly 15x the throughput of batch size 1. The trade-off is latency — batch size 16 introduces a queuing delay of up to several seconds depending on request arrival rate. For applications where users expect sub-second responses (chat, voice, search), real-time or low-latency batching (dynamic batching with small batches) is necessary. For offline processing (data enrichment, content generation, bulk analysis), large-batch inference is the most impactful cost-saving technique available.

Model Quantization: INT8 vs INT4 vs FP8

Quantization reduces the number of bits used to represent each model weight, directly reducing memory footprint and memory bandwidth requirements. On NVIDIA H100 GPUs, INT8 tensor cores provide 2x the throughput of FP16, and INT4 provides 4x. The practical impact on cost-per-token is roughly proportional to the bit reduction: INT4 models deliver approximately 4x more tokens per dollar than FP16 models.

FP8 — The safe default

FP8 quantization is natively supported on H100 GPUs and causes effectively zero accuracy degradation for models up to 405B parameters. FP8 uses 8 bits with a floating-point representation (1 sign, 4 exponent, 3 mantissa for E4M3 format), preserving dynamic range better than integer formats. For teams starting their optimization journey, FP8 is the recommended first step — it delivers a 2x memory reduction and throughput improvement with no measurable accuracy loss and minimal engineering effort [6].

INT8 — The workhorse

INT8 quantization with techniques like SmoothQuant or LLM.int8() achieves 4x memory reduction compared to FP32 while maintaining accuracy within 0.5-1% of the original model. On H100 GPUs, INT8 tensor cores provide roughly 2x the throughput of FP16. The practical cost reduction is approximately 3x compared to FP16 inference (2x from computation speedup, 1.5x from ability to fit larger models on fewer GPUs). INT8 is the most widely deployed quantization format in production as of 2026 [4].

INT4 — Maximum compression

INT4 quantization via GPTQ or AWQ achieves 8x memory reduction compared to FP32. A 70B model that requires 140 GB in FP16 can run in 35 GB with INT4, fitting on a single A100 (80 GB) or even an RTX 4090 (24 GB) with KV cache pressure management. The accuracy impact is 1-3% on most benchmarks for large models (70B+), but can reach 4-6% for smaller models (7B-13B). For high-volume, cost-sensitive workloads where some accuracy degradation is acceptable, INT4 provides the lowest cost-per-token available [7].

For a detailed technical comparison of quantization techniques, see our guide on LLM quantization and model compression.

Speculative Decoding

Speculative decoding accelerates inference by using a small, fast draft model to propose multiple candidate tokens, which the large target model then verifies in parallel. Because the verification pass processes all candidate tokens in a single forward pass (rather than generating them one-by-one), speculative decoding achieves 2-3x throughput improvement for the same hardware cost [8].

The effectiveness of speculative decoding depends on the acceptance rate — the fraction of draft tokens that the target model accepts. Acceptance rates of 70-90% are typical when the draft model is well-matched to the target model (e.g., a distilled version of the same architecture). Acceptance rates drop to 30-50% when the draft model is architecturally dissimilar. Speculative decoding is most effective for latency-sensitive applications where batching is not feasible, as it provides throughput improvement without adding latency.

Practical implementations in vLLM and TensorRT-LLM support speculative decoding out of the box. For a typical deployment, adding a 1B-parameter draft model alongside a 70B target model reduces cost-per-token by approximately 40-50%, with no impact on output quality since the target model's distribution is preserved.

KV Cache Optimization

The KV cache is the single largest memory consumer in production LLM deployments with long contexts or high concurrency. For a 70B model generating 4K-token outputs across 100 concurrent conversations, the KV cache consumes 200 GB — more than the 140 GB required for model weights. KV cache optimization directly impacts both cost (fewer GPUs needed) and throughput (more requests per GPU).

KV cache quantization

Quantizing the KV cache from FP16 to FP8 reduces memory consumption by 50% with negligible accuracy impact for most models. INT4 KV cache quantization achieves 75% memory reduction with 0.5-1% accuracy degradation on long-context tasks. vLLM and TensorRT-LLM support KV cache quantization natively [3].

Prefix caching

When multiple requests share common prefixes (system prompts, context documents), prefix caching stores the KV cache entries for the shared prefix once and reuses them across requests. For RAG applications where every request includes the same set of context documents, prefix caching reduces effective KV cache memory by 60-80% and eliminates the computation for the shared prefix, reducing time-to-first-token by 50-70% [9]. See our guide on RAG best practices for detailed implementation patterns.

PagedAttention

PagedAttention, the core innovation behind vLLM, manages KV cache memory in fixed-size blocks rather than contiguous allocations, eliminating fragmentation and enabling memory sharing across requests. In production deployments, PagedAttention improves KV cache memory utilization from approximately 40% (naive contiguous allocation) to 85-90%, effectively increasing request throughput per GPU by 2-2.5x without additional hardware [3].

Batching Strategies

The choice of batching strategy has a 3-10x impact on cost-per-token. Three primary approaches dominate production deployments:

Dynamic batching

The simplest approach: accumulate requests for a fixed time window (typically 100-500ms) or until a minimum batch size is reached, then process them together. Dynamic batching is easy to implement and reduces latency variance compared to static batching. The main limitation is that requests arriving after the window closes must wait for the next batch, introducing latency jitter. Throughput improvement over single-request inference is typically 3-5x [5].

Continuous batching (in-flight batching)

Rather than waiting for all sequences in a batch to complete, continuous batching evicts finished sequences mid-iteration and inserts new ones. This eliminates the wasted computation of waiting for the slowest sequence in each batch — a problem known as "straggler effect" that grows worse with output length variance. Continuous batching improves throughput by 20-40% over dynamic batching for realistic request distributions where output lengths vary widely. vLLM, TensorRT-LLM, and TGI all implement continuous batching [3].

Disaggregated serving

The most advanced approach separates prefill (context processing) and decode (token generation) onto different GPU pools. Prefill is compute-bound (takes roughly 10-50ms for a 2K-token input) while decode is memory-bandwidth-bound (takes roughly 10-30ms per token). By dedicating GPU resources to each phase, disaggregated serving achieves 30-60% higher throughput for mixed workloads. This approach is supported in production by TensorRT-LLM and is increasingly adopted at scale [2].

Model Distillation for Cost Reduction

Knowledge distillation trains a smaller student model to replicate the behavior of a larger teacher model. The result is a permanently smaller model that requires no special infrastructure — a distilled 7B model can run on a single consumer GPU while matching the output quality of a 70B model on specific tasks.

The cost equation is compelling: a distilled 7B model running on an RTX 4090 generates tokens at approximately $0.05 per million tokens (GPU amortization and power), compared to $3-10 per million tokens for frontier API models. For high-volume workloads, distillation can reduce inference costs by 50-100x [10].

The key constraint is that distillation is task-specific. A distilled model trained on summarization tasks will not perform well at code generation. Teams must invest in task-specific distillation datasets and evaluation. The upfront cost of distillation (typically $10,000-50,000 in compute for a 7B student) is recovered quickly at scale — typically within 1-3 months of production deployment at volumes exceeding 10 million tokens per day.

For guidance on when to invest in distillation vs using existing models, see our comparison of open-source vs closed-source LLMs.

Provider Cost Comparison

Inference costs vary dramatically across providers and deployment models. The choice between API-based consumption and self-hosted infrastructure depends on volume, latency requirements, and operational capacity.

Pricing as of June 2026. Self-hosted costs include GPU instance costs and power. Serverless providers include profit margins above raw compute cost. These price points suggest a 5-30x cost range between the cheapest and most expensive deployment options [1][11].

Monitoring Cost-Per-Query

You cannot optimize what you do not measure. Cost-per-query should be a first-class metric in any production AI system, tracked alongside latency and quality metrics. The key metrics are:

• Cost per generation — total compute cost divided by number of generations. Target: track by model, by use case, and by prompt template.
• Cost per token — total compute cost divided by output tokens generated. Target: under $1.00 per million tokens for production systems.
• GPU utilization — percentage of theoretical GPU compute time actually used for generation. Target: above 60% for batch workloads, above 20% for real-time workloads.
• KV cache hit rate — percentage of tokens served from prefix cache rather than recomputed. Target: above 50% for RAG workloads [9].

Most cost optimization opportunities surface through monitoring: a prompt template that uses 8K tokens of context when 2K would suffice, a model that is 10x more expensive than necessary for a given task, a batching configuration that leaves GPUs idle 80% of the time. Instrumentation is the first step toward optimization.

For a broader framework on measuring production AI systems, see our guide on LLM evaluation and production metrics.

The Optimization Stack: Putting It All Together

The most cost-effective deployments combine multiple optimization techniques. The compounding effect is substantial: quantization (4x), continuous batching (1.5x), prefix caching (2x), and speculative decoding (2x) together achieve approximately 24x cost reduction compared to a naive single-request FP16 deployment.

The recommended optimization order is:

1. Quantize to FP8 (2x improvement, zero engineering effort)
2. Implement continuous batching via vLLM (1.5x, framework migration)
3. Enable prefix caching (1.5-2x, application-level changes)
4. Quantize KV cache to FP8 (1.2x, configuration change)
5. Add speculative decoding (2x, draft model deployment)
6. Quantize weights to INT4 for high-volume pathways (2x, model conversion)
7. Distill task-specific student models (2-5x, training investment)

Each step requires different levels of engineering investment. The first three steps deliver roughly 6x cost reduction with standard frameworks and moderate engineering effort. The remaining steps deliver additional gains but require dedicated ML infrastructure expertise.

For guidance on the infrastructure decisions behind these choices, see our complete guide to AI infrastructure and our analysis of parallel processing and GPU architectures.