CNN Architecture Explained: A Complete Guide from the Ground Up

Introduction

In 1989, Yann LeCun published a paper demonstrating that a neural network with a constrained architecture — one that forced each neuron to connect only to a small local region of the input — could recognise handwritten digits with remarkable accuracy. That architecture, later refined into LeNet-5, was the first convolutional neural network. It was, in essence, an attempt to hard-wire the inductive biases of vision into a learnable system.

Thirty-seven years later, CNNs have evolved from a niche technique for postal code recognition into the bedrock of computer vision. They power medical image diagnosis, autonomous vehicle perception, satellite imagery analysis, facial recognition systems, and industrial quality inspection. Even as Transformers challenge their dominance, CNNs remain the most computationally efficient and widely deployed architecture for visual understanding.

This guide explains CNN architecture from first principles. We cover every layer type — convolution, pooling, activation, normalisation, and fully connected — walk through the forward pass in detail with code, and examine the design principles that separate effective architectures from ineffective ones. By the end, you will have the tools to design, analyse, and debug CNNs from scratch.

What Makes a CNN Different

A standard fully connected network treats each input pixel as an independent feature. For a 224x224 RGB image, that means 150,528 input values. A single hidden layer with 1,024 neurons would require over 150 million parameters — and that is just one layer. Such networks overfit catastrophically on images, generalise poorly, and ignore the spatial structure that defines visual data.

CNNs solve this with three architectural innovations:

• Local connectivity: Each neuron connects only to a small spatial region of the input, not the full image. This reflects the biological fact that visual neurons respond to stimuli in restricted receptive fields.
• Weight sharing: The same set of weights (a kernel or filter) is applied across every spatial location in the image. This dramatically reduces parameters and ensures translation invariance — a cat detected in the top-left corner is also detected in the bottom-right.
• Hierarchical feature learning: Early layers detect simple patterns (edges, corners, textures). Deeper layers compose those into mid-level features (eyes, wheels, windows). The deepest layers learn high-level semantic concepts (faces, cars, buildings).

These three properties — local connectivity, weight sharing, and hierarchical composition — define the CNN design philosophy. Every architectural decision flows from them.

For a refresher on the underlying linear algebra, see our post on vectors, tensors, and scalars in AI.

The Convolution Layer

The convolution layer is the core building block of a CNN. It applies a set of learnable kernels (filters) across the input, producing a set of feature maps that encode the presence of specific visual patterns at each spatial location.

Kernels and Filters

A kernel is a small matrix of learnable weights, typically 3x3, 5x5, or 7x7. Each kernel detects a specific pattern. A 3x3 kernel has 9 weight parameters plus one bias term. A collection of kernels operating on the same input is called a filter bank.

The operation itself is straightforward: slide the kernel across the input in steps (stride), at each position compute the element-wise product between the kernel and the overlapping input region, sum the products, and add a bias. The result at each position forms one element of the output feature map.

The Convolution Operation

For a single-channel 2D input I and a kernel K, the output feature map S at position (i, j) is:

S[i, j] = (I * K)[i, j] = sum_m sum_n I[i+m, j+n] * K[m, n] + b

Where * denotes the convolution operator (technically cross-correlation, as deep learning frameworks skip the kernel flip of true convolution). The sum runs over the kernel dimensions m and n.

In practice, inputs are multi-channel. A colour image has 3 channels (RGB). A hidden layer might have 64, 128, or 256 channels. In the multi-channel case, each kernel is a 3D tensor with depth equal to the input's channel count. The convolution sums across both spatial dimensions and the channel dimension:

S_f[i, j] = sum_c sum_m sum_n I[c, i+m, j+n] * K_f[c, m, n] + b_f

Each filter f produces one output channel. A layer with 64 filters produces a 64-channel output feature map.

import numpy as np

def conv2d_forward(image, kernel, stride=1, padding=0):
    """
    Perform a 2D convolution on a single-channel image.
    
    image: (H, W) input image
    kernel: (kH, kW) convolution kernel
    stride: step size for sliding the kernel
    padding: zero-padding size around the image
    """
    if padding > 0:
        image = np.pad(image, padding, mode='constant')
    
    H, W = image.shape
    kH, kW = kernel.shape
    
    out_h = (H - kH) // stride + 1
    out_w = (W - kW) // stride + 1
    
    output = np.zeros((out_h, out_w))
    
    for i in range(out_h):
        for j in range(out_w):
            h_start = i * stride
            h_end = h_start + kH
            w_start = j * stride
            w_end = w_start + kW
            receptive_field = image[h_start:h_end, w_start:w_end]
            output[i, j] = np.sum(receptive_field * kernel)
    
    return output

For deeper coverage of the convolution operation itself, including dilated convolution, grouped convolution, and separable convolution, see our dedicated post on the mathematics of convolution.

Spatial Dimensions: Stride, Padding, and Kernel Size

Three hyperparameters control the spatial dimensions of the output feature map:

• Kernel size (k): The spatial extent of the kernel, typically odd (3, 5, 7) to ensure symmetric padding around the centre pixel.
• Stride (s): The step size between consecutive kernel positions. Stride 1 produces a densely computed output. Stride 2 down-samples the feature map by roughly half.
• Padding (p): Zero-pixels added around the input boundary. “Same” padding (p = (k - 1) / 2) preserves spatial dimensions. “Valid” padding (p = 0) reduces them.

The output spatial size for a given input size H is:

H_out = (H_in + 2p - k) / s + 1

Understanding this equation is essential for designing CNN architectures. If you chain many convolution and pooling layers, the spatial dimensions shrink progressively — you need to ensure they never go negative or fractional.

Receptive Fields

The receptive field of a neuron is the region of the original input that influences its activation. A single 3x3 convolution has a receptive field of 3x3. Stack two 3x3 convolutions, and the effective receptive field grows to 5x5. Stack three, and it is 7x7.

This is the principle of hierarchical composition. Early layers see tiny patches of the image (edges, corners). As layers stack, the receptive field expands to cover larger structures (textures, shapes, objects). The rate of receptive field growth is a critical design consideration. Using larger kernels (7x7) grows the field faster, but at higher computational cost. Stacking smaller kernels (3x3) grows it more slowly but with fewer parameters and more non-linearity between layers.

Activation Functions in CNNs

Convolution is a linear operation. Without non-linear activation functions between layers, a CNN would collapse into a single linear transformation, losing all representational power.

The most common activation in early CNNs was the sigmoid or tanh, but these suffer from the vanishing gradient problem: for large positive or negative inputs, the gradient approaches zero, effectively halting learning. Modern CNNs use:

• ReLU (Rectified Linear Unit): f(x) = max(0, x). The default choice since AlexNet (2012). Computes instantly, produces sparse activations, and mitigates vanishing gradients for positive values. The downside: “dead ReLU” units that never activate and never learn.
• Leaky ReLU: f(x) = max(alpha * x, x) with alpha typically 0.01. Passes a small gradient for negative values, preventing dead units. Used in some object detection architectures.
• ELU and SELU: Exponential variants that smooth the transition near zero, providing better gradient flow and self-normalising properties in deep networks.
• Swish / SiLU: f(x) = x * sigmoid(x). Discovered through architecture search. Produces smoother optimisation landscapes than ReLU and is used in EfficientNet and ConvNeXt.

In practice, ReLU remains the most common choice for CNNs due to its simplicity and speed. Swish offers marginal accuracy improvements at the cost of slightly more computation.

Pooling Layers

Pooling layers reduce the spatial dimensions of feature maps, providing two benefits: computational efficiency (fewer parameters in subsequent layers) and translation invariance (small spatial shifts in the input produce the same pooled output).

Max Pooling

Max pooling is the dominant pooling operation. A 2x2 max pooling window slides over the feature map with stride 2, outputting the maximum value within each window. This halves the spatial dimensions: a 224x224 feature map becomes 112x112.

The intuition is that the maximum activation represents the strongest response to a detected pattern. The exact location of that pattern within the window is discarded — the network cares that an edge is present, not whether it was positioned 2 pixels left or right.

def max_pool2d(image, pool_size=2, stride=None):
    """
    Max pooling operation.
    
    image: (C, H, W) input tensor
    pool_size: size of the pooling window
    """
    if stride is None:
        stride = pool_size
    
    C, H, W = image.shape
    out_h = (H - pool_size) // stride + 1
    out_w = (W - pool_size) // stride + 1
    
    output = np.zeros((C, out_h, out_w))
    
    for c in range(C):
        for i in range(out_h):
            for j in range(out_w):
                h_start = i * stride
                w_start = j * stride
                window = image[c, h_start:h_start+pool_size, w_start:w_start+pool_size]
                output[c, i, j] = np.max(window)
    
    return output

Average Pooling and Global Pooling

Average pooling computes the mean of the window values instead of the maximum. It preserves more information about weaker activations but is less effective at capturing the presence of specific features. Average pooling is sometimes used in the final layers of an architecture to replace fully connected layers (global average pooling), which reduces parameters and overfitting.

Global average pooling, introduced in Network in Network (2013), computes the mean of each entire feature map, producing a single value per channel. This creates a fixed-size output regardless of input resolution, enabling the network to process images of arbitrary size.

Strided Convolution vs Pooling

Modern architectures increasingly replace pooling layers with strided convolutions (convolution with stride 2). A strided convolution simultaneously learns which information to keep while down-sampling, rather than applying a fixed rule like max. This trend began with the all-convolutional network (Springenberg et al., 2014) and is now standard in architectures like ResNet and ConvNeXt. However, max pooling remains useful for its strong translation invariance and zero-parameter cost.

Normalisation Layers

Training deep CNNs is notoriously sensitive to the distribution of layer activations. If activations grow or shrink across layers, gradients can vanish or explode. Normalisation layers address this by stabilising the distribution of activations.

Batch Normalisation

Batch normalisation (Ioffe and Szegedy, 2015) was a breakthrough that enabled training of much deeper networks. It normalises each channel's activations across the batch dimension, subtracting the batch mean and dividing by the batch standard deviation. Two learnable parameters (gamma and beta) then scale and shift the normalised values.

y = gamma * (x - mu_batch) / sqrt(sigma_batch^2 + epsilon) + beta

Batch norm provides several benefits: it allows higher learning rates (by preventing activations from growing too large), reduces sensitivity to weight initialisation, and provides a mild regularisation effect (because each batch's statistics introduce noise).

The trade-off: batch norm behaves differently at training and inference time. At inference, it uses running averages of the batch statistics rather than the actual batch statistics, which can cause discrepancies when batch size varies. It also requires sufficiently large batch sizes to produce stable statistics, which can be problematic for high-resolution inputs or large models.

Layer, Instance, and Group Normalisation

Alternative normalisation strategies exist for scenarios where batch norm falls short:

• Layer normalisation: Normalises across all channels and spatial locations for each sample independently. Standard in Transformers, less common in CNNs.
• Instance normalisation: Normalises per channel per sample. Used in style transfer and image generation tasks.
• Group normalisation: Splits channels into groups and normalises within each group. A practical middle ground that works well with small batch sizes, popularised in computer vision by the Detectron2 framework.

Fully Connected Layers and the Classifier Head

After a stack of convolution and pooling layers has transformed the input into high-level feature maps, a classifier head maps these features to the final output (typically class probabilities). The classifier usually consists of one or more fully connected layers followed by a softmax or sigmoid.

The transition from feature maps to the classifier requires flattening the spatial dimensions. If the final feature map has shape (C, H, W), flattening produces a vector of length C * H * W. This vector is then fed through the fully connected layers.

Several design decisions matter here:

• Global average pooling before FC: Instead of flattening, apply global average pooling to produce a C-dimensional vector. This eliminates the dependence on input resolution and drastically reduces parameters. Used in ResNet and most modern architectures.
• Dropout: A regularisation technique that randomly sets a fraction of neurons to zero during training. Standard dropout rates are 0.5 for the first FC layer and 0.2-0.3 for subsequent ones. Dropout is less critical when global average pooling is used, but remains common.
• Number of FC layers: AlexNet used three FC layers (9x6x6 -> 4096 -> 4096 -> 1000). Modern architectures typically use one or two, relying on the convolutional backbone for feature extraction.

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(128 * 4 * 4, 256),
            nn.ReLU(inplace=True),
            nn.Linear(256, num_classes),
        )
    
    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

The Complete Forward Pass: A Worked Example

Let us trace a 224x224 RGB image through a typical CNN to see how the dimensions evolve at each layer. This is a simplified ResNet-style architecture:

• Input: (3, 224, 224) — 3 colour channels, 224x224 pixels.
• Conv1: 64 filters of size 7x7, stride 2, padding 3. Output: (64, 112, 112). Spatial size halves: (224 + 6 - 7) / 2 + 1 = 112.
• MaxPool: 3x3 window, stride 2, padding 1. Output: (64, 56, 56).
• Conv2_x: Three residual blocks, each with 64 filters, 3x3 kernels, stride 1, padding 1. Output: (64, 56, 56). Spatial dimensions preserved.
• Conv3_x: Four residual blocks, 128 filters, first block uses stride 2. Output: (128, 28, 28).
• Conv4_x: Six residual blocks, 256 filters, first block uses stride 2. Output: (256, 14, 14).
• Conv5_x: Three residual blocks, 512 filters, first block uses stride 2. Output: (512, 7, 7).
• Global Average Pooling: Output: (512, 1, 1), flattened to a 512-dimensional vector.
• FC: 512 -> 1000. Output: 1000 class logits.

The total reduction in spatial dimensions is 224 / 7 = 32x (achieved through stride-2 operations at Conv1, the pooling layer, and the transitions between residual stages). The channel count increases from 3 to 512, capturing progressively more abstract features at each scale.

Design Principles for CNN Architectures

Decades of empirical research have distilled several principles for designing effective CNN architectures:

Depth vs Width

Deeper networks with more layers generally outperform wider networks with more channels per layer, for the same parameter budget. The intuition: depth enables hierarchical feature composition, while width primarily increases representational capacity at a single scale. He et al. (2016) showed that a 152-layer ResNet outperforms a shallower but wider network with the same parameter count. This drove the trend from VGG (16-19 layers) to ResNet (50-152 layers) to DenseNet and ResNeXt.

Bottleneck Designs

A bottleneck layer compresses the channel dimension before applying expensive 3x3 convolutions, then expands it back. A ResNet bottleneck is: 1x1 conv (reduce channels from 256 to 64), 3x3 conv (spatial processing), 1x1 conv (expand channels back to 256). This reduces the computational cost of a 3x3 convolution by 4x while maintaining representational capacity.

Information Flow and Skip Connections

Skip connections (residual connections) allow gradients to flow directly through the network by adding the input of a block to its output. This became essential for training very deep networks (50+ layers), as it provides a “gradient highway” that prevents signal decay. DenseNet extended this idea by connecting each layer to every subsequent layer, maximising information flow at the cost of increased memory.

Parameter Efficiency

The number of parameters in a convolution layer is: (k_h * k_w * C_in + 1) * C_out. For a 3x3 convolution with 256 input and output channels, that is (9 * 256 + 1) * 256 = 590,080 parameters. Depthwise separable convolution (used in MobileNet and EfficientNet) factorises this into a depthwise convolution (256 * 9 parameters) and a pointwise convolution (256 * 256 parameters), reducing the total to approximately 68,000 — a 8.7x reduction with minimal accuracy loss.

For a deeper comparison of architectural evolution and parameter efficiency, see our post on the evolution of CNN architectures from LeNet to ConvNeXt.

CNN vs MLP: A Parameter Comparison

To appreciate the efficiency of CNNs, consider what a fully connected network would require to process the same 224x224 RGB image. A single hidden layer with 1,024 neurons would have 3 * 224 * 224 * 1024 = 154 million parameters. A CNN achieving comparable capacity with 64 feature maps, four 3x3 convolution layers, and a classifier head uses roughly 1-2 million parameters — a 100x reduction.

This efficiency is not a free lunch. CNNs hard-code spatial locality and translation invariance into their architecture, which is excellent for images but inappropriate for data without spatial structure. For tabular data, MLPs still outperform CNNs. For sequential data, RNNs or Transformers are more natural choices.

Training Considerations

Training a CNN requires careful management of several interacting factors:

• Weight initialisation: Proper initialisation prevents vanishing or exploding gradients in the early stages of training. He initialisation (Kaiming He et al., 2015) is standard for ReLU-based CNNs. It sets the variance of weights to 2 / (k_h * k_w * C_in), ensuring that the variance of activations is preserved through the forward pass.
• Learning rate schedule: A common strategy is to start with a relatively high learning rate (0.1 for ImageNet training) and decay it by a factor of 10 at predetermined epochs (30, 60, 80). Cosine annealing schedules, which smoothly decay the learning rate following a cosine curve, have become increasingly popular as they require fewer hyperparameter choices.
• Data augmentation: CNNs are data-hungry. Random horizontal flips, random crops, colour jitter, and mixup augmentation can effectively multiply the dataset size and prevent overfitting. Augmentation is often more impactful than architectural changes for improving generalisation.
• Regularisation: Weight decay (L2 regularisation) is universally applied, typically with a coefficient between 1e-4 and 1e-5. Dropout is used in the classifier head. Label smoothing (replacing hard 0/1 targets with soft targets like 0.1/0.9) reduces overconfidence and improves calibration.

Common Pitfalls and Debugging Strategies

When designing and training CNNs, several issues arise frequently:

• Dimension mismatch: The most common error. Always compute output dimensions layer by layer. One wrong padding or stride value and the dimensions go negative or produce fractions.
• Dead ReLU: If a layer produces only negative values after initialisation, ReLU kills all gradients and the layer never learns. Reduce the learning rate or switch to Leaky ReLU.
• Overfitting: Training loss decreases but validation loss increases. Add dropout, increase weight decay, add data augmentation, or reduce model capacity.
• Underfitting: Both training and validation loss are high. Increase model capacity (more layers or channels), reduce regularisation, or check for data issues.
• Training instability with batch norm: Small batch sizes produce noisy batch statistics. Use group normalisation or increase batch size.