How Large Language Models Work: From Training to Inference

Introduction

Large language models like GPT-4o, Claude 4, Gemini 2.0, and Llama 3 have become ubiquitous in 2026, but their internal workings remain opaque to most practitioners. This article provides a clear, structured explanation of how LLMs work across the three stages of their lifecycle: pre-training, fine-tuning, and inference. We cover tokenization, the transformer architecture, training objectives, what model parameters actually store, and the mechanics of text generation.

This guide is designed for engineers and technical product managers who work with LLMs but want a deeper understanding of how they function. We avoid unnecessary mathematical formalism while maintaining technical accuracy, and we connect each concept to its practical implications for building AI applications.

The Three Stages of an LLM's Lifecycle

Every LLM goes through three distinct stages that determine what it knows and how it behaves:

Stage 1: Pre-training

The model is trained on a massive corpus of text (2-15 trillion tokens for frontier models) to predict the next token in a sequence. This stage imparts language understanding, factual knowledge, and reasoning patterns. Pre-training is the most computationally expensive stage, costing $50-500 million for state-of-the-art models as of 2026 [1]. The output of this stage is a base model that can complete text but has not been taught to follow instructions or answer questions.

Stage 2: Fine-tuning

The base model is further trained on curated datasets of instruction-response pairs, typically using a combination of supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). This stage teaches the model to follow instructions, provide helpful responses, refuse harmful requests, and maintain conversation structure. Fine-tuning costs $1-10 million for frontier models and transforms the base model into a useful assistant [2].

Stage 3: Inference

The trained model is deployed to generate responses to user inputs. Given a prompt, the model generates one token at a time in an autoregressive loop, each step conditioned on all previously generated tokens. Inference is the operational stage — the model no longer learns or updates its parameters. The quality of inference is determined by the model's training and by the inference configuration (temperature, top-p, system prompt) [3].

Tokenization: How Text Becomes Numbers

Before an LLM can process text, the text must be converted into a sequence of numbers. This is the job of the tokenizer. A tokenizer maps text to tokens — discrete units that are typically 3-5 characters for English text but can be sub-word units, whole words, or individual characters depending on the tokenization algorithm.

The dominant tokenization algorithm in 2026 is Byte-Pair Encoding (BPE), which starts with individual characters and iteratively merges the most frequent adjacent pairs into new tokens [4]. For example, the word "understanding" might be tokenized as ["under", "standing"] if the training corpus shows that "understanding" is not common enough to warrant its own token. The word "the" almost always receives a single token because it appears so frequently.

The vocabulary size — the total number of distinct tokens — varies by model. GPT-4o uses a vocabulary of approximately 100,000 tokens. Llama 3 uses 128,000 tokens. The tokenizer is learned during pre-training and remains fixed for the lifetime of the model. Tokenizer quality has a meaningful impact on model efficiency: a model with a vocabulary that tokenizes text efficiently (more information per token) requires fewer tokens to represent the same input, reducing both training and inference costs.

For a deeper exploration of how tokenization relates to vector representations, see our guide on vectors, tensors, and scalars in AI.

The Transformer Architecture

The transformer architecture, introduced by Vaswani et al. in 2017, is the foundation of all modern LLMs [5]. A transformer consists of a stack of identical layers (typically 32-120 layers for frontier models), each containing two main components: multi-head self-attention and feed-forward neural networks.

Self-attention

Self-attention is the mechanism that allows each token to "look at" every other token in the input. For each token, the model computes a weighted sum of the representations of all other tokens, where the weights depend on the relevance between tokens. In the sentence "The cat sat on the mat because it was comfortable," self-attention allows the model to determine that "it" refers to "the mat" based on the contextual relationship between the tokens.

Multi-head attention runs this process in parallel across multiple "heads" (typically 32-96 heads), each learning to attend to different types of relationships — syntactic, semantic, positional, etc. The outputs of all heads are concatenated and projected back to the model's hidden dimension. This is the key mathematical operation that makes transformers so effective at capturing long-range dependencies in text.

Feed-forward networks

After self-attention, each token's representation passes through a feed-forward network (FFN) — typically a two-layer MLP with a hidden dimension 4x the model dimension. The FFN applies the same transformation to each token independently, projecting the representation into a higher-dimensional space and back. This is where the model stores most of its factual knowledge: the FFN weights encode patterns that the model has learned from the training data [6].

Layer normalization and residual connections

Each sub-layer (attention and FFN) is wrapped with a residual connection (adding the input back to the output) and layer normalization (stabilizing activations). The residual connections are critical — they allow gradients to flow through the entire depth of the network during training, enabling effective learning in models with 100+ layers. Without residual connections, training deep transformers would be practically impossible due to vanishing gradients [5].

For a detailed walkthrough of the transformer architecture with diagrams, see our dedicated guide on the transformer architecture explained.

Training: Next Token Prediction

The training objective for LLMs is deceptively simple: given a sequence of tokens, predict the next token. The model processes the input sequence through its transformer layers, producing a probability distribution over the entire vocabulary for the next token position. The loss function is cross-entropy between the predicted distribution and the actual next token.

During training, the model processes sequences of tokens (typically 4,096-131,072 tokens, depending on the model's context window) and computes the loss for each token's prediction. The gradients from all token positions are averaged, and the model's parameters are updated via backpropagation. This is repeated for billions of training steps across trillions of tokens.

The critical insight is that next-token prediction is a remarkably effective training signal. To predict the next token accurately, the model must learn grammar (to know which word forms are correct), factual knowledge (to know which entity is likely in context), reasoning patterns (to track logical relationships), and discourse structure (to maintain coherent narrative). The model does not need explicit supervision for any of these capabilities — they emerge as byproducts of the token prediction task [7].

However, next-token prediction also has limitations. It encourages the model to produce plausible continuations rather than factually correct ones — a key cause of hallucination. See our exploration of AI hallucination causes and prevention for a deeper discussion of this training objective mismatch.

What Do Model Parameters Store?

Each parameter in an LLM is simply a number — a floating-point value that is learned during training and fixed during inference. With 70 billion to over 1 trillion parameters in frontier models, what do all these numbers actually represent?

The answer is that parameters encode patterns at multiple levels of abstraction. Lower layers (closer to the input) learn syntactic and surface-level patterns: how words combine into phrases, grammatical roles, basic word relationships. Middle layers learn semantic and factual patterns: entity relationships, conceptual hierarchies, domain knowledge. Higher layers (closer to the output) learn abstract patterns: reasoning strategies, task structures, stylistic conventions, and instruction-following behavior [8].

An important concept is that parameters do not store discrete facts in a structured database sense. There is no entry that says "Paris is the capital of France." Instead, the model's knowledge of this fact is distributed across millions of parameters that collectively encode a representation space where the relationship between France and Paris is captured. This distributed representation is what makes LLMs robust to variation (they can answer "What is the capital of France?" and "Tell me about Paris, the capital city of France" using the same distributed knowledge) and also what makes them prone to hallucination (they can produce plausible but incorrect associations).

During fine-tuning, parameters are adjusted to align the model's behavior with human preferences. For a detailed guide on how fine-tuning works, see our LLM fine-tuning guide.

Inference: How LLMs Generate Text

During inference, the model generates text one token at a time in an autoregressive loop. The process has several components:

The autoregressive loop

Given the input prompt, the model computes a probability distribution over the vocabulary for the next token. A token is sampled from this distribution (see sampling strategies below), appended to the input sequence, and the process repeats. The generation continues until an end-of-sequence token is produced or a maximum length is reached.

This sequential generation is the fundamental performance bottleneck for LLMs. Each token requires a full forward pass through the transformer — roughly 700 billion FLOPs for a 70B model. A 1,000-token response requires 700 trillion FLOPs, which takes roughly 30-60 seconds on an H100 GPU at 30-50% utilization. The sequential nature of generation is why inference remains expensive and latency-sensitive despite dramatic hardware improvements [9].

The KV cache

The KV cache is a critical optimization that makes autoregressive generation practical. During generation, the model computes Key and Value matrices for each token as part of the self-attention mechanism. These matrices represent the "keys" that tokens use to attend to earlier tokens (the values are the representations of earlier tokens themselves).

Without a KV cache, the model would recompute the Key and Value matrices for every previous token at each generation step — making the total computation scale as O(n^2) where n is the sequence length. With a KV cache, the Key and Value matrices for tokens up to position t-1 are stored in memory and reused when generating token t. This reduces the computation to O(n) per token — a dramatic improvement that makes long-form generation feasible [10].

The KV cache is also the dominant memory consumer during inference for long sequences. For a 70B model with a 128K-token context window, the KV cache can consume 200+ GB of GPU memory. For this reason, KV cache optimization (quantization, paging, prefix caching) is a major focus of inference engine development. See our guide on LLM inference cost optimization for details.

Temperature and top-p sampling

The model's final layer produces logits — unnormalized scores — for each token in the vocabulary. These logits are converted to probabilities via the softmax function. Two key parameters control the randomness of token selection:

• Temperature divides the logits by the temperature value before softmax. Temperature 0 (greedy decoding) always selects the highest-probability token, producing deterministic but potentially repetitive outputs. Temperature 1.0 samples from the full distribution, maximizing diversity. Typical production values are 0.1-0.7 for factual tasks and 0.7-1.0 for creative tasks.
• Top-p (nucleus sampling) selects from the smallest set of tokens whose cumulative probability exceeds p. For example, top-p = 0.9 selects from the top tokens that together account for 90% of the probability mass, excluding the long tail of low-probability tokens. This prevents sampling from very unlikely tokens while maintaining diversity among plausible ones.

For practical guidance on configuring these parameters for production applications, see our prompt engineering guide for production.

Scaling Laws: Why Bigger Models Work Better

The empirical observation that drives the entire field of large language models is that model performance improves predictably as model size, dataset size, and compute increase — following power-law relationships known as scaling laws [11].

The key finding from Kaplan et al. (2020) is that loss decreases as a power law with model size, dataset size, and compute, with no signs of diminishing returns at the scales tested. Each 10x increase in model parameters yields a consistent reduction in loss, and the relationship holds across orders of magnitude. This finding has driven the trend toward ever-larger models: from GPT-3 (175B parameters in 2020) to GPT-4 (rumored ~1.8T parameters in 2023) to models exceeding 1T parameters in 2026.

The Chinchilla scaling law (Hoffmann et al., 2022) refined this understanding by showing that model size and training data must be scaled together. For a given compute budget, there is an optimal ratio of model parameters to training tokens — approximately 20 tokens per parameter. A 70B model should be trained on roughly 1.4 trillion tokens. Training a smaller model on more data (or a larger model on less data) is suboptimal for the same compute budget [12].

The practical implications are significant. Scaling laws mean that the primary determinant of model quality is compute investment — there are no architectural shortcuts that can match the gains from simply training a larger model on more data. However, scaling laws also mean that smaller, well-trained models can achieve surprising capability. A 7B model trained on Chinchilla-optimal data can match a 70B model trained on insufficient data for the same tasks.

For evaluating whether bigger models are worth the additional cost for your application, see our guide on LLM evaluation and production metrics.

Putting It All Together: A Complete Example

Here is how all the components work together when you send a prompt to an LLM:

1. The prompt text is tokenized into a sequence of token IDs (e.g., "What is the capital of France?" becomes [4123, 84, 592, 1042, 437, 2341, 31])
2. Each token ID is mapped to its embedding vector — a learned representation in the model's hidden dimension space (typically 4,096-16,384 dimensions)
3. Positional embeddings are added to encode the position of each token in the sequence
4. The embedded sequence passes through the transformer layers. At each layer, self-attention computes relationships between all tokens, and feed-forward networks transform each token's representation
5. The final layer produces logits — scores for every token in the vocabulary
6. Logits are converted to probabilities via temperature-adjusted softmax
7. A token is sampled using top-p or top-k sampling from the probability distribution
8. The sampled token is appended to the sequence, and steps 4-7 repeat (using the KV cache for efficiency) until an end-of-sequence token is reached
9. The generated token IDs are detokenized back into human-readable text

Each step in this process involves mathematical operations on massive matrices, but the conceptual flow is straightforward: convert text to numbers, process through stacked pattern-recognition layers, and decode the result back to text.