Diffusion Models: Architecture, Training, and Applications

Introduction

In 2021, two papers from OpenAI and UC Berkeley rewrote the playbook for generative modelling. Denoising Diffusion Probabilistic Models (DDPM) demonstrated that a simple process — gradually adding noise to data and learning to reverse it — could generate images competitive with Generative Adversarial Networks (GANs). Then, in 2022, Latent Diffusion (Stable Diffusion) showed that running this process in a compressed latent space reduced inference cost by an order of magnitude, making high-quality image generation accessible on consumer GPUs.

By 2026, diffusion models have become the dominant generative paradigm across image, video, audio, and 3D content creation. They power tools like Midjourney, DALL-E 3, Stable Diffusion 3, Sora, and dozens of open-source variants. Unlike GANs, which require careful balancing of generator and discriminator training, diffusion models optimise a simple regression objective and produce remarkably stable training dynamics. Unlike autoregressive models, they can generate high-resolution outputs at any aspect ratio without sequential token prediction overhead.

This guide provides a complete technical walkthrough of diffusion models. We cover the mathematical foundations, the U-Net architecture and its innovations, latent diffusion mechanics, sampling strategies and their trade-offs, conditioning modalities from text-to-image to ControlNet, extensions to video and 3D, and practical considerations for production deployment. By the end, you will understand how diffusion models work under the hood and how to build systems around them.

How Diffusion Works

Diffusion models are inspired by non-equilibrium thermodynamics. The core idea is simple: define a process that gradually destroys structure in data by adding noise, then learn a reverse process that reconstructs the data from noise. If the forward process is a Markov chain that adds Gaussian noise over T steps, the reverse process learns to denoise step by step, eventually recovering a sample from the training distribution.

Forward Diffusion Process

Given a data point x₀ sampled from the training distribution, the forward process produces a sequence x₁, x₂, ..., x_T by adding Gaussian noise at each timestep. At each step t, the transition is:

q(x_t | x_t-1) = N(x_t; √(1 - β_t) x_t-1, β_t I)

The variance schedule β₁, ..., β_T controls how much noise is added at each step. In the original DDPM, β increases linearly from 10^-4 to 0.02 over T = 1,000 steps. The key property of the forward process is that we can sample x_t directly from x₀ in closed form:

q(x_t | x₀) = N(x_t; √(ᾱ_t) x₀, (1 - ᾱ_t) I)

where α_t = 1 - β_t and ᾱ_t = ∏_s=1^t α_s. At sufficiently large T, x_T approaches an isotropic Gaussian distribution, completely destroying the original data structure.

def forward_diffusion(x_0, noise_schedule, T):
    """
    Add Gaussian noise to an image over T timesteps.
    
    x_0: clean input image
    noise_schedule: beta_t values for each timestep
    T: number of diffusion steps
    """
    x_t = x_0
    for t in range(1, T + 1):
        beta_t = noise_schedule[t]
        epsilon = torch.randn_like(x_t)
        x_t = torch.sqrt(1 - beta_t) * x_t + torch.sqrt(beta_t) * epsilon
    return x_t

Reverse Denoising Process

The reverse process learns to invert the forward diffusion. Starting from pure noise x_T ~ N(0, I), we apply a learned transition p_θ(x_t-1 | x_t) at each step, gradually removing noise to recover a clean sample. When the noise added at each forward step is small (β_t is small), the reverse transition can be parameterised as a Gaussian:

p_θ(x_t-1 | x_t) = N(x_t-1; μ_θ(x_t, t), σ_t² I)

Instead of predicting μ_θ directly, Ho et al. (2020) found it is more effective to predict the noise ε added at each step, then derive μ_θ from the reparameterisation. This reparameterisation is the key insight that makes diffusion training tractable.

class NoisePredictor(nn.Module):
    def __init__(self, unet):
        super().__init__()
        self.unet = unet
    
    def denoise_step(self, x_t, t, conditioning=None):
        """
        Predict noise and remove one step.
        x_t: noisy image at timestep t
        t: current timestep
        """
        predicted_noise = self.unet(x_t, t, conditioning)
        alpha_t = 1 - beta_t
        alpha_bar_t = torch.cumprod(alpha_t, dim=0)
        mu = (1 / torch.sqrt(alpha_t)) * (
            x_t - (beta_t / torch.sqrt(1 - alpha_bar_t)) * predicted_noise
        )
        if t > 0:
            z = torch.randn_like(x_t)
        else:
            z = 0
        return mu + torch.sqrt(beta_t) * z

For readers new to the underlying neural network architectures that power these models, see our guides on CNN architecture and transformer architecture.

Training Objective

The training objective for diffusion models is surprisingly simple. Ho et al. (2020) showed that a simplified loss function — mean squared error between the true noise and the predicted noise — works as well as the full variational lower bound:

L_simple = E_{t, x0, ε} [ || ε - ε_θ(x_t, t) ||² ]

Where t is uniformly sampled from {1, ..., T}, x_t is obtained by adding noise ε ~ N(0, I) to x₀, and ε_θis the neural network's prediction. The network learns to predict the noise component at step t given the noisy image and the timestep.

This objective has a deep connection to score-based generative modelling. The score function ∇_x log p(x) points in the direction of increasing data density. Song and Ermon (2019) showed that estimating the score is equivalent to denoising — the optimal way to remove noise from a corrupted sample is to follow the score. In diffusion models, the predicted noise ε_θ is directly proportional to the score: ∇_x log p(x) ∝ -ε_θ(x_t, t) / σ_t. This score-matching perspective unified several lines of research and led to the continuous-time formulation of score-based diffusion models (Song et al., 2021).

In practice, the loss is often weighted by the signal-to-noise ratio at each timestep. Noisy steps (high t) receive lower weight because the signal is already corrupted, while early steps (low t) receive higher weight because the network needs to be precise about fine details. The original DDPM uses uniform weighting, but v-prediction (Salimans and Ho, 2022) and min-SNR weighting (Hang et al., 2023) improve sample quality by rebalancing these contributions.

Architectural Innovations

The neural network that predicts noise — typically a U-Net — has evolved significantly since the original DDPM. Modern diffusion backbones incorporate several architectural innovations that improve both generation quality and computational efficiency.

U-Net Backbone

The standard architecture for image diffusion models is a U-Net: an encoder-decoder network with skip connections between corresponding resolution levels. The encoder downsamples the input through a series of convolutional blocks, capturing increasingly abstract features at lower resolutions. The decoder upsamples back to the original resolution, using skip connections from the encoder to recover spatial details lost during downsampling.

A typical diffusion U-Net operates at four resolution levels (e.g., 64, 32, 16, 8 at the smallest). Each level contains 2-3 residual blocks with convolutional layers, group normalisation, and SiLU activations. The base channel dimension is typically 128 or 256, doubling at each downsampling step and halving during upsampling. For Stable Diffusion 3, the U-Net has approximately 860 million parameters in its largest configuration.

Cross-Attention for Conditioning

The key innovation that enabled text-to-image generation was the introduction of cross-attention layers in the U-Net. At each resolution level, the spatial feature maps attend to a sequence of text embeddings produced by a CLIP or T5 text encoder. The cross-attention mechanism follows the standard transformer formulation:

Attention(Q, K, V) = softmax(Q K^T/ √d) V

Where Q is derived from the spatial feature map (the U-Net hidden state), and K and V are derived from the text embedding sequence. This allows each spatial position to selectively attend to relevant words in the prompt — "a red car on a mountain road" activates different attention maps for "car" and "mountain."

For a deeper dive into how attention mechanisms work, see our guide on transformer architecture.

Timestep Embeddings

The diffusion model needs to know which timestep it is at. Timestep information is injected via sinusoidal position encodings — the same formulation used in the original transformer paper (Vaswani et al., 2017) — or learned embedding tables. These are added to the U-Net's residual block outputs through adaptive group normalisation (AdaGN), where the timestep embedding modulates the scale and shift parameters of each normalisation layer:

AdaGN(h, t) = t_scale * GroupNorm(h) + t_shift

This conditioning mechanism ensures that the network behaves differently at different noise levels — at high noise levels it focuses on structure, while at low noise levels it refines details.

Sinusoidal Position Encodings

The timestep t is encoded using sinusoidal functions of different frequencies:

PE_{(t, 2i)} = sin(t / 10000^2i/d)
PE_{(t, 2i+1)} = cos(t / 10000^2i/d)

Where d is the embedding dimension (typically 256 or 512) and i indexes the embedding dimension. The varying frequencies allow the model to distinguish between adjacent timesteps (high frequencies) while maintaining long-range coherence across the full diffusion timeline (low frequencies). This encoding is passed through an MLP (typically 2-3 linear layers with SiLU activation) before being injected into the U-Net blocks.

Latent Diffusion (Stable Diffusion)

The original DDPM operates directly on pixels. For a 512x512 RGB image, the U-Net must process 786,432 dimensions at every resolution level. This is computationally prohibitive — training requires hundreds of GPU-days and inference takes tens of seconds even on high-end hardware.

Rombach et al. (2022) introduced Latent Diffusion Models (LDM), which perform the diffusion process in a compressed latent space learned by a Variational Autoencoder (VAE). This is the architecture behind Stable Diffusion and its successors.

VAE Compression: Pixel Space to Latent Space

A VAE is trained to compress images into a lower-dimensional latent representation and reconstruct them with high fidelity. The encoder maps a 512x512x3 image to a 64x64x4 latent tensor — a 48x reduction in spatial dimensions and a 192x reduction in total elements (786,432 pixels to 16,384 latent values). The decoder learns the inverse mapping.

The VAE is trained with a combination of reconstruction loss (L1 or perceptual LPIPS loss), KL divergence to regularise the latent distribution, and a discriminator loss (from GAN training) to ensure sharp reconstructions. Stable Diffusion uses a VAE with a downsampling factor f = 8, producing latents at 1/8 the spatial resolution of the input. The KL penalty weight is set to 10^-6, which produces a near-deterministic encoder with minimal noise in the latent space.

Denoising in Latent Space

Once the VAE is trained and frozen, the diffusion U-Net operates entirely in the 64x64x4 latent space. The forward and reverse processes are identical but applied to latents instead of pixels. The U-Net's computational cost drops by roughly the compression factor: processing 16,384 elements instead of 786,432. This reduces training time from thousands of GPU-days to hundreds, and inference time from tens of seconds to 2-5 seconds on an RTX 4090.

# Simplified Latent Diffusion pipeline
class LatentDiffusion(nn.Module):
    def __init__(self, vae, unet, text_encoder):
        super().__init__()
        self.vae = vae              # Pre-trained VAE
        self.unet = unet            # Denoising U-Net
        self.text_encoder = text_encoder  # CLIP text encoder
    
    def encode(self, x):
        # Compress pixels to latent space
        with torch.no_grad():
            return self.vae.encode(x).latent_dist.sample() * 0.18215
    
    def decode(self, z):
        # Reconstruct latent to pixel space
        with torch.no_grad():
            return self.vae.decode(z / 0.18215).sample
    
    def forward(self, latent, timestep, text_embed):
        # Denoise in latent space
        return self.unet(latent, timestep, encoder_hidden_states=text_embed)

Text Conditioning with CLIP

Stable Diffusion uses a frozen CLIP text encoder (ViT-L/14, 427M parameters) to convert text prompts into a sequence of 77 token embeddings, each 768 dimensions. These embeddings are fed into the U-Net's cross-attention layers as described above. The choice of CLIP is important: its contrastive training objective aligns text and image embeddings in a shared space, meaning the text encoder produces embeddings that are naturally "image-aware."

In Stable Diffusion 3 and Flux, the text conditioning has been upgraded to use T5-XXL (11B parameters) alongside CLIP, providing richer understanding of complex prompts, negations, and fine-grained attributes. The trade-off is increased memory and inference latency — a T5 forward pass adds roughly 100ms to each generation step.

For more on how multimodal embeddings bridge text and vision, see our guide on multimodal AI.

Sampling Strategies

The original DDPM requires 1,000 sequential denoising steps to generate a single image. This is impractically slow for real-time applications. A rich body of research has developed faster sampling methods that produce high-quality images in 1-50 steps.

DDIM: Denoising Diffusion Implicit Models

DDIM (Song et al., 2021) reformulates the reverse process as a non-Markovian chain, allowing the model to skip steps during sampling. Instead of denoising at every trained timestep, DDIM defines a new reverse process that only evaluates a subset of steps (e.g., 50 steps instead of 1,000). The key insight is that the forward process can be made deterministic given x₀ and x_t, enabling "implicit" sampling that skips intermediate steps. DDIM produces 20x faster sampling than DDPM with comparable quality, making it the default sampler in most diffusion-based tools.

DPM-Solver and DPM-Solver++

DPM-Solver (Lu et al., 2022, arXiv:2206.00927) treats the diffusion ODE (ordinary differential equation) with a high-order numerical solver. It exploits the fact that the reverse diffusion process can be expressed as an exponentially weighted integral of the score function. By approximating this integral with a high-order Taylor expansion, DPM-Solver achieves high-quality generation in 10-20 steps — roughly 50-100x faster than the original DDPM. DPM-Solver++ extends this with adaptive step size selection and improved handling of the sampling trajectory's curvature.

Euler Ancestral Sampler

The Euler ancestral sampler is a simple first-order ODE solver with stochastic noise injection at each step. It approximates the reverse SDE (stochastic differential equation) by taking a deterministic Euler step followed by adding random noise scaled by the step size. Euler ancestral is the default sampler in ComfyUI and many community tools because it balances speed and quality without requiring complex solver implementations. It typically produces good results in 20-30 steps.

Classifier-Free Guidance

Classifier-free guidance (CFG), introduced by Ho and Salimans (2022), is the standard technique for controlling how strongly the generated image adheres to the conditioning prompt. The idea is to interpolate between the conditional prediction (with prompt) and the unconditional prediction (without prompt, or with a null text embedding):

ε_θ^CFG = ε_θ(x_t, t, ∅) + w * (ε_θ(x_t, t, c) - ε_θ(x_t, t, ∅))

Where w is the guidance scale. At w = 1, the prediction is purely conditional. At w = 7, the conditional signal is amplified 7x, producing images that more closely match the prompt but can become over-saturated or unnatural. Typical CFG values range from 3 to 14, with 7 being a common default. Higher guidance scales improve prompt adherence but reduce diversity and image quality — this is the guidance- diversity trade-off.

Sampling Method Comparison

The table below summarises the key characteristics of each sampling method.

For production systems where latency matters, DPM-Solver++ at 15 steps with CFG scale 7 is the recommended starting point. For maximum quality (e.g., final rendered assets), DDIM at 100 steps with CFG scale 5 produces slightly better results at the cost of 6x longer generation time.

Conditioning Modalities

Diffusion models are remarkably flexible about what they can condition on. While text-to-image is the most famous application, the same underlying architecture supports diverse conditioning modalities.

Text-to-Image

The canonical application. A text prompt is encoded via CLIP or T5, and the embeddings condition the U-Net through cross-attention. State-of-the-art models in 2026 — Stable Diffusion 3.5, Flux.1, and Midjourney v7 — operate at resolutions up to 2048x2048 with native multi-aspect-ratio support. Flux.1 uses a flow-matching objective with 12 billion parameters and generates 1024x1024 images in 4 steps via rectified flow.

Image-to-Image and Inpainting

Image-to-image generation conditions on both a text prompt and an input image. The input image is partially noised to a specific timestep, and the reverse process reconstructs it with modifications guided by the prompt. The noise level determines how much the output deviates from the input — denoising from t = 200 (out of 1,000) preserves most of the original structure, while t = 800 allows dramatic changes.

Inpainting is a special case where a binary mask specifies regions to regenerate. The masked areas are filled with noise, and the unmasked areas are blended with the original image at each denoising step. Stable Diffusion's inpainting pipeline uses a concatenated input of the noisy image, the mask, and the original image (with masked regions zeroed), feeding 7 input channels instead of the standard 4 (latent).

ControlNet

ControlNet (Zhang et al., 2023, arXiv:2302.05543) adds spatial conditioning signals — edge maps, depth maps, pose skeletons, normal maps, segmentation masks — to the diffusion process without modifying the base model. A trainable copy of the U-Net encoder is locked to the conditioning input (e.g., a Canny edge map), and its features are added to the main U-Net's decoder via zero-initialised convolution layers.

The "zero convolution" innovation is critical: ControlNet layers are initialised with zeros so they produce no effect at the start of training, allowing the base model's weights to remain untouched. Training only the ControlNet parameters (roughly 10-15% of the total model size) requires 200-500 GPU-hours on a single A100, compared to thousands of hours for full fine-tuning. This has led to a rich ecosystem of ControlNet variants for Canny edge, HED boundary, depth (MiDaS), normal maps, OpenPose skeletons, and M-LSD line detection.

LoRA for Personalization

Low-Rank Adaptation (LoRA), originally developed for LLMs, has been adapted for diffusion models as a lightweight personalisation technique. A LoRA layer adds a low-rank update to the cross-attention and feed-forward weights of the U-Net. Training a LoRA on a specific subject (a person, a product, an art style) requires only 100-500 images and completes in 10-30 minutes on a single GPU.

Multiple LoRAs can be composed during inference: one LoRA for the subject, another for the artistic style, and a third for the lighting condition. The composability of LoRAs — enabled by their linear additive updates — makes them the standard mechanism for personalisation in the open-source diffusion ecosystem. The Civitai platform hosts over 500,000 community-trained LoRAs.

For a detailed comparison of LoRA and other parameter-efficient fine-tuning methods, see our guide on LLM fine-tuning (the same principles apply to diffusion models).

Video and 3D Diffusion

Extending diffusion from single images to video and 3D content introduces fundamental challenges around temporal consistency, spatial coherence across views, and computational scaling.

Video Diffusion Models

Video diffusion extends the 2D U-Net to 3D by adding a time (frame) dimension. The architecture becomes a 3D U-Net with spatiotemporal convolutions: 3D convolutions replace 2D convolutions, operating over both spatial and temporal axes. Each residual block processes a 5D tensor of shape (batch, frames, channels, height, width).

Temporal attention layers are inserted between spatial attention layers to model frame-to-frame correspondences. These are typically causal or bidirectional attention over the frame dimension, allowing the model to learn motion patterns, object persistence, and camera movement. The joint training objective is a simple extension of the image diffusion loss — predict noise for all frames simultaneously:

L_video = E [ || ε - ε_θ(x_t^1:N, t) ||² ]

Where x_t^1:N denotes N video frames at noise level t. Training requires approximately 10-30x more compute than image diffusion: a 16-frame 256x256 video has 4x the pixel count of a single image, and the 3D convolutions add additional overhead.

Frame Consistency

The primary failure mode of video diffusion is flickering — objects appearing, disappearing, or changing appearance between frames. Several techniques address this:

• Temporal conditioning: Each frame's denoising step receives the previous frame's latent as additional input, enforcing temporal smoothness.
• Masked noise: During training, a random subset of frames receives higher noise levels, forcing the model to learn frame interpolation and inpainting.
• Hierarchical generation: Generate a low-resolution video first, then upsample frame-by-frame with temporal consistency constraints.
• Recurrent refinement: Run multiple diffusion passes where the output of the first pass conditions the second, similar to how a video codec uses motion vectors between frames.

Sora and the Transformer Paradigm Shift

OpenAI's Sora (2024) introduced a fundamentally different architecture for video generation. Instead of a 3D U-Net, Sora uses a diffusion transformer (DiT) operating on spacetime patches. Videos are divided into patches of fixed spatial and temporal extent (e.g., 16x16x4 pixels/tokens), similar to how ViT patches images. The patches are flattened into a sequence and processed by a standard transformer with self-attention.

This design has several advantages. Transformers handle variable-length sequences naturally, so Sora generates videos at arbitrary resolutions, durations, and aspect ratios without architecture changes. The self-attention mechanism captures long-range dependencies across both space and time, eliminating the limited receptive field of 3D convolutions. Sora ultimately operates at scale (reportedly 10-20 billion parameters) and generates videos up to 60 seconds with remarkable consistency.

The DiT architecture has since been adopted for image generation as well — Stable Diffusion 3 and Flux both use transformer-based backbones, signalling a convergence of image and video architectures toward patch-based diffusion transformers.

3D Generation

3D diffusion operates on volumetric representations like NeRFs (Neural Radiance Fields), triplanes (three orthogonal feature planes popularised by EG3D), or 3D Gaussian splats. The diffusion process adds noise to the 3D representation itself — for NeRFs, noise is added to the density and colour values at each sampled point along a ray.

Zero-1-to-3 (Liu et al., 2023) finetunes a 2D diffusion model to generate novel views conditioned on a single input image, effectively turning a 2D model into a 3D-aware one without explicit 3D training data. Score Distillation Sampling (SDS, Poole et al., 2022) goes further: it uses a pre-trained 2D diffusion model as a critic to optimise a 3D representation (e.g., a NeRF or mesh) by backpropagating the diffusion loss through a differentiable renderer. This produces coherent 3D objects from text prompts but requires 1-2 hours of optimisation per object.

Production Deployment

Deploying diffusion models in production requires careful management of compute resources, latency budgets, and safety requirements. Unlike LLM inference, which is memory-bandwidth-bound due to autoregressive token generation, diffusion inference is compute-bound — the U-Net forward pass through multiple resolution levels is a dense computation that fully utilises GPU tensor cores.

Compute Requirements and GPU Memory

A single Stable Diffusion 3.5 inference pass at 1024x1024 with 28 denoising steps, DPM-Solver++ sampler, and CFG scale 7 requires:

• GPU memory: 8-12 GB for the U-Net (2.6B parameters in bf16), 6 GB for the VAE, 3 GB for the CLIP text encoder, and 2-4 GB for activations — total ~20-24 GB for batch size 1.
• Inference latency: 3-5 seconds on an RTX 4090 (24 GB), 1.5-2.5 seconds on an H100 (80 GB), 6-10 seconds on an A10G (24 GB, standard AWS instance).
• Throughput: ~12-20 images per minute on a single H100 at batch size 1. Batching to batch size 4 increases throughput to ~35-50 images per minute with proportional memory increase.

Batching and Optimisation

Several techniques reduce inference cost in production:

• Model distillation: Progressive distillation (Salimans and Ho, 2022) trains a student model to match the output of a teacher model in fewer sampling steps, reducing step count by 2-4x with minimal quality loss.
• VAE caching: The VAE encoder/decoder is used once per generation. Precomputing and caching VAE outputs for common input sizes avoids redundant computation.
• TensorRT compilation: Compiling the U-Net with TensorRT reduces inference latency by 30-50% through layer fusion, kernel autotuning, and FP16/INT8 precision.
• Scheduler optimisation: Replacing the standard DDPM scheduler with DPM-Solver++ at 15 steps (instead of 28) reduces compute by 46% with negligible quality impact for most prompts.

Safety Filters and NSFW Detection

Production diffusion deployments must implement safety guardrails at multiple stages:

• Input prompt filtering: Blocklist-based filtering with semantic similarity to known unsafe concepts. OpenAI's Moderation API or a fine-tuned BERT classifier can flag prompts requesting violent, sexual, or hateful content before inference begins.
• Latent inspection: Some implementations inspect the latent tensor after the final denoising step but before VAE decoding, checking for statistical anomalies that correlate with NSFW content (e.g., unusual activation patterns in skin-colour regions).
• Output classification: A dedicated NSFW classifier (e.g., CLIP-based safety checker or a fine-tuned ViT) evaluates the generated image and blocks or blurs flagged outputs. Stable Diffusion's safety checker uses a cosine similarity threshold between the generated image embedding and a set of unsafe concept embeddings.
• Watermarking: Invisible watermarking embedded in the VAE decoder's output allows provenance tracking. Google's SynthID and Stable Signature (Fernandez et al., 2023) embed imperceptible patterns that survive cropping, resizing, and JPEG compression.

For infrastructure considerations around deploying generative AI at scale, see our guide on AI infrastructure and edge AI deployment.