CNN Architectures: From LeNet to ConvNeXt — The Complete Evolution

Introduction

The history of convolutional neural network architecture is a story of progressive refinement. Each generation solved a specific problem that limited the previous one — vanishing gradients, parameter inefficiency, computational cost, or deployment constraints — and in doing so, revealed new challenges for the next. From LeNet-5's humble 60,000 parameters to ConvNeXt's hundreds of millions, the evolution spans three decades of research.

This post traces that lineage, examining each major architecture, the problem it solved, the innovations it introduced, and the principles it established. Understanding this history provides a mental model for designing new architectures and choosing the right one for a given task.

LeNet-5: The Original (1998)

Yann LeCun's LeNet-5 established the template that all subsequent CNNs follow: alternating convolution and pooling layers, followed by fully connected layers. It was designed for handwritten digit recognition (MNIST) and used by American banks to process cheques. The architecture had only 7 layers and 60,000 parameters. The first convolution used 5x5 kernels with stride 1, producing 6 feature maps. A 2x2 average pooling halved the spatial dimensions.

LeNet introduced ideas that remain standard: local receptive fields, weight sharing, and spatial subsampling. However, it used sigmoid/tanh activations (which limited depth due to vanishing gradients) and average pooling (later supplanted by max pooling). Most importantly, LeNet was constrained by the compute of its era — training took days on CPUs.

AlexNet: The Deep Learning Breakthrough (2012)

Alex Krizhevsky's AlexNet won the 2012 ImageNet challenge by a dramatic margin, reducing top-5 error from 26% to 15%. This is widely regarded as the moment that kicked off the modern deep learning revolution. AlexNet was 8 layers with 60 million parameters. Key innovations included ReLU activations (non-saturating non-linearity that accelerated training), dropout (regularisation by randomly dropping 50% of neurons), data augmentation (random crops and flips), GPU training (split across two GPUs), and overlapping max pooling.

AlexNet's success demonstrated that depth and scale matter — deeper networks with more parameters, trained on more data with more compute, produce dramatically better results. This set the stage for the scaling race that continues today.

VGGNet: The Power of Depth (2014)

VGGNet showed that simply stacking more 3x3 convolutions produced better results than AlexNet's varied kernel sizes. VGG-16 (16 weight layers) and VGG-19 became standard backbones. The key insight: two 3x3 convolutions have the same receptive field as one 5x5 but with fewer parameters (18 vs 25) and more non-linearity. Three 3x3 replace a 7x7 with even greater savings (27 vs 49).

VGG's simplicity was its strength, but it was computationally expensive — VGG-16 required 15.5 GFLOPs per forward pass, most consumed by the first FC layer (102 million of 138 million total parameters). This inefficiency drove the search for more parameter-efficient designs.

Inception (GoogLeNet): Multi-Scale Processing (2014)

The Inception architecture won ImageNet 2014 with a different philosophy: instead of going deeper, process at multiple scales simultaneously. The Inception module applied 1x1, 3x3, and 5x5 convolutions plus 3x3 max pooling in parallel, concatenating all outputs.

The genius was using 1x1 convolutions as bottleneck layers before expensive 3x3 and 5x5 convolutions. A 1x1 convolution with fewer output channels compresses the channel dimension, reducing computational cost by 4-10x. This allowed Inception to be 12x more parameter-efficient than AlexNet while being significantly deeper (22 layers). Subsequent versions (v2, v3, v4) added batch norm, factorised convolutions, label smoothing, and residual connections.

ResNet: Residual Learning (2015)

Residual Networks solved the degradation problem that had limited network depth. Counter-intuitively, adding more layers increased training error, suggesting that optimising very deep networks was fundamentally difficult. ResNet's solution was elegant: add skip connections that bypass one or more layers, forcing the network to learn the residual mapping F(x) = H(x) - x. If the optimal mapping is the identity, the network pushes residuals towards zero, which is much easier than learning the identity through non-linear layers.

class ResidualBlock(nn.Module):
    def __init__(self, in_c, out_c, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, 3, stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_c)
        self.conv2 = nn.Conv2d(out_c, out_c, 3, stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_c)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_c != out_c:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_c, out_c, 1, stride=stride, bias=False),
                nn.BatchNorm2d(out_c))
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        return F.relu(out)

ResNet-152 achieved 3.57% top-5 error on ImageNet — surpassing human-level performance (estimated at 5%) for the first time. It became the standard backbone for nearly all computer vision tasks. The bottleneck block uses 1x1 convolutions to reduce and restore the channel dimension, making 3x3 convolutions much cheaper.

class Bottleneck(nn.Module):
    def __init__(self, in_c, mid_c, out_c, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, mid_c, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(mid_c)
        self.conv2 = nn.Conv2d(mid_c, mid_c, 3, stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(mid_c)
        self.conv3 = nn.Conv2d(mid_c, out_c, 1, bias=False)
        self.bn3 = nn.BatchNorm2d(out_c)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_c != out_c:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_c, out_c, 1, stride=stride, bias=False),
                nn.BatchNorm2d(out_c))
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = F.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        out += self.shortcut(x)
        return F.relu(out)

DenseNet: Maximum Information Flow (2017)

DenseNet took skip connections to the logical extreme: connect each layer to every subsequent layer. The input to each layer is the concatenation of all feature maps from preceding layers. This maximises information flow and alleviates the vanishing gradient problem even more effectively than ResNet. DenseNet is parameter-efficient despite having many connections because each layer can be very thin (e.g., 12 filters). DenseNet-121 uses 8 million parameters while matching ResNet-50's accuracy (25 million parameters).

The trade-off is memory consumption during training. DenseNet must cache all intermediate feature maps for the backward pass, which grows quadratically with depth. Several memory-efficient implementations mitigate this, but it remains a practical limitation.

ResNeXt: Cardinality as a Design Dimension (2017)

ResNeXt introduced grouped convolutions into the ResNet framework, adding cardinality as a third design dimension alongside depth and width. Cardinality refers to the number of parallel transformation paths within a block. The critical finding: increasing cardinality is more effective than increasing depth or width for the same parameter budget. A ResNeXt-101 with 32 groups outperforms a deeper ResNet-200 while using fewer parameters.

MobileNet and EfficientNet: Efficiency-First Design (2017-2019)

MobileNet introduced depthwise separable convolutions as the basic building block, reducing computational cost by 8-9x with minimal accuracy loss. MobileNetV2 introduced inverted residual blocks with linear bottlenecks (expand the channels instead of compressing like ResNet). MobileNetV3 used neural architecture search (NAS) to optimise the layer configuration.

EfficientNet systematised the trade-off between depth, width, and resolution through compound scaling — scaling all three dimensions by fixed factors. EfficientNet-B0 was discovered through NAS, and larger variants (B1-B7) were generated by applying the scaling rule. EfficientNet-B7 achieved ImageNet top-1 accuracy of 84.4% with 10x fewer parameters than previous state-of-the-art models.

ConvNeXt: Modernising the CNN (2022)

In 2022, as Vision Transformers challenged CNN dominance, Liu et al. published ConvNeXt: a pure CNN that matches Swin Transformer performance through targeted modernisations applied to ResNet. Changes include: training on ImageNet-21K pretraining (ViT recipe), patchify stem (4x4 stride-4 conv similar to ViT patch embedding), inverted bottleneck (hidden dimension 4x input), 7x7 depthwise convolutions (larger receptive fields), GELU activations, fewer normalisation layers, LayerNorm instead of BatchNorm, and separate down-sampling layers.

ConvNeXt demonstrates that CNNs, when properly modernised, can match Transformer performance on vision tasks. This ended the CNN vs Transformer debate — both architectures are viable, and the choice depends on practical considerations like latency, memory, and hardware utilisation.

Design Principles Across Eras

• Deep and thin beats shallow and wide: For a fixed parameter budget, deeper networks with fewer channels per layer outperform shallower networks with more channels.
• Bottleneck designs improve efficiency: Compress the channel dimension before expensive spatial convolutions, then expand again. Used in ResNet, Inception, MobileNetV2, and ConvNeXt.
• Skip connections enable depth: Every architecture with more than 30 layers uses some form of residual connection.
• Normalisation is essential: Batch norm (or variants) is critical for training stability. The shift from BN to LN in ConvNeXt reflects Transformer influence.
• Larger receptive fields help: From 11x11 in AlexNet to 7x7 depthwise in ConvNeXt, the trend is towards more global context.
• Training recipe matters as much as architecture: Much of ConvNeXt's improvement comes from better training techniques rather than architectural changes alone.