Object Detection with CNNs: From R-CNN to YOLO

Introduction

Image classification answers the question "what is in this image?" Object detection answers a harder question: "what objects are in this image, and where exactly are they located?" It requires a CNN to produce both class labels and bounding box coordinates for every object instance in the image, regardless of the number of objects.

This additional requirement — detecting an arbitrary number of objects at varying scales and positions — makes object detection fundamentally more challenging than classification. The field has produced two families of approaches: two-stage detectors (R-CNN family) that first propose regions and then classify them, and one-stage detectors (YOLO, SSD) that predict boxes and classes directly in a single pass.

This guide traces the evolution of object detection architectures, from the slow but accurate R-CNN to the blazing-fast YOLO and the balanced RetinaNet. We cover the core concepts — region proposals, anchor boxes, non-maximum suppression, and detection-specific loss functions — that underpin all modern detectors.

R-CNN: Regions with CNN Features (2014)

The original R-CNN (Girshick et al., 2014) was the first deep learning approach to achieve significant improvements over traditional computer vision methods. It worked in three stages. First, selective search generated approximately 2,000 region proposals per image (candidate bounding boxes likely to contain objects). Second, each proposal was warped to a fixed size (227x227) and passed through a CNN (AlexNet) to extract a feature vector. Third, class-specific SVMs classified each feature vector, and a bounding box regressor refined the box coordinates.

R-CNN was a breakthrough but had crippling practical limitations. Processing 2,000 proposals per image meant 2,000 forward passes through a CNN per image. Training required fine-tuning the CNN on warped proposals, training 200 SVMs (one per class), and training bounding box regressors — a multi-stage pipeline that was slow to train and slow to inference (47 seconds per image on a GPU).

Fast R-CNN: Shared Computation (2015)

Fast R-CNN addressed R-CNN's inefficiency with a single key insight: instead of running the CNN 2,000 times per image, run it once on the entire image and share the feature maps across all proposals. A Region of Interest (RoI) pooling layer extracted fixed-size feature maps from arbitrary-sized region proposals by dividing each proposal into a grid (e.g., 7x7) and max-pooling each grid cell.

The architecture was a single network: the input image passed through a CNN backbone (e.g., VGG-16), producing a shared feature map. Selective search proposals were projected onto this feature map. RoI pooling extracted a fixed-size feature vector for each proposal. Two sibling output layers followed: a softmax classifier (K+1 classes, including background) and a bounding box regressor (4*K outputs for class-specific box refinement).

Fast R-CNN trained end-to-end with a multi-task loss: L = L_cls + lambda * L_box. The classification loss was standard cross-entropy. The box regression loss used smooth L1, which is less sensitive to outliers than L2. Training was 9x faster than R-CNN, and inference was 146x faster (0.32 seconds per image). However, selective search remained the bottleneck — it could not run on a GPU and added 2 seconds per image.

Faster R-CNN: Learnable Region Proposals (2016)

Faster R-CNN eliminated the selective search bottleneck by introducing the Region Proposal Network (RPN) — a fully convolutional network that predicts region proposals directly from the feature map. The RPN slides a small network over the feature map, at each position predicting k anchor boxes of different scales and aspect ratios. For each anchor, it outputs an objectness score (is this an object or not?) and 4 bounding box regression offsets.

The full Faster R-CNN pipeline: CNN backbone produces a feature map; RPN processes it to produce region proposals; RoI pooling extracts features for each proposal; classifier and regressor heads produce final predictions. The entire system is a single, differentiable network trained end-to-end with four losses (RPN classification, RPN regression, Fast R-CNN classification, Fast R-CNN regression).

Faster R-CNN achieved near real-time inference (5-17 fps depending on the backbone) with state-of-the-art accuracy. It became the de facto standard for object detection, with numerous extensions (Mask R-CNN for instance segmentation, Cascade R-CNN for higher quality detection, FPN for multi-scale detection).

YOLO: You Only Look Once (2016)

YOLO (Redmon et al., 2016) took a radically different approach: treat detection as a single regression problem. Divide the image into an SxS grid. Each grid cell predicts B bounding boxes (each with 5 values: x, y, w, h, confidence) and C class probabilities. The output is an S x S x (B*5 + C) tensor, processed in a single forward pass.

YOLO's loss function combines coordinate regression, confidence prediction, and classification into a single sum-of-squared-errors objective:

def yolo_loss(predictions, targets, num_boxes=7, num_classes=20):
    """
    Simplified YOLO loss function.
    predictions: (S, S, B*5 + C) tensor
    targets: (S, S, B*5 + C) ground truth
    """
    S, S, _ = predictions.shape
    coord_loss = 0
    conf_loss = 0
    class_loss = 0
    
    for i in range(S):
        for j in range(S):
            # Find the best bounding box (highest IoU with ground truth)
            best_box = 0
            best_iou = 0
            for b in range(B):  # B boxes per grid cell
                iou = compute_iou(predictions[i,j,b*5:b*5+4], targets[i,j,b*5:b*5+4])
                if iou > best_iou:
                    best_iou = iou
                    best_box = b
            
            # Coordinate loss (only for responsible box)
            box_pred = predictions[i, j, best_box*5:best_box*5+4]
            box_true = targets[i, j, best_box*5:best_box*5+4]
            coord_loss += lambda_coord * sum((box_pred - box_true)**2)
            
            # Confidence loss
            conf_pred = predictions[i, j, best_box*5+4]
            conf_true = targets[i, j, best_box*5+4]
            conf_loss += (conf_pred - conf_true)**2
            
            # Class loss (if object present)
            if targets[i, j, B*5:] > 0:
                class_pred = predictions[i, j, B*5:]
                class_true = targets[i, j, B*5:]
                class_loss += sum((class_pred - class_true)**2)
    
    return coord_loss + conf_loss + class_loss

YOLO was revolutionary for its speed — 45 fps for the base model and 155 fps for Fast YOLO. The trade-off was lower accuracy than Faster R-CNN, particularly for small objects and objects in crowded scenes (because each grid cell could only predict one class, and the spatial resolution was limited).

YOLO Evolution: v2 through v10

YOLOv2 (YOLO9000) introduced anchor boxes, batch normalisation, and joint training on detection and classification datasets (9,000 classes). YOLOv3 added a feature pyramid network (FPN) backbone for multi-scale detection, significantly improving small object detection. YOLOv4 incorporated the Bag of Freebies (BoF) and Bag of Specials (BoS) design philosophies — data augmentation, Mish activation, CIoU loss, and PANet neck — achieving a perfect balance of speed and accuracy.

YOLOv5-v10 refined the architecture further with automated hyperparameter optimisation, model scaling (n/s/m/l/x variants), Focus and C3 modules, decoupled detection heads, and task-specific loss functions. Ultralytics YOLOv8, released in 2023, became the most widely deployed detection framework, supporting detection, segmentation, classification, and pose estimation in a single codebase.

SSD: Single Shot MultiBox Detector (2016)

SSD (Liu et al., 2016) combined YOLO's single-shot approach with Faster R-CNN's anchor box concept. The key innovation was multi-scale detection: SSD made predictions from feature maps at six different resolutions (from 38x38 down to 1x1). Earlier (larger) feature maps detected small objects; later (smaller) feature maps detected large objects. Each feature map location predicted a fixed set of anchor boxes at different scales and aspect ratios.

SSD achieved YOLO-like speed (59 fps for SSD300) with Faster R-CNN-like accuracy (74.3% mAP on VOC2007). The main weakness was detecting very small objects, since the early high-resolution feature maps had limited semantic information. This was addressed in later work by adding feature fusion connections (DSSD, FSSD).

RetinaNet and Focal Loss (2017)

RetinaNet (Lin et al., 2017) addressed a fundamental problem with one-stage detectors: the extreme class imbalance between foreground (object) and background (non-object) examples. In a typical training image, there might be 100,000 anchor boxes but only 1-10 containing objects. The vast number of easy background examples overwhelms the standard cross-entropy loss, preventing the model from learning to discriminate hard foreground examples.

The solution was focal loss: FL(p_t) = -(1 - p_t)^gamma * log(p_t). The modulating factor (1 - p_t)^gamma down-weights easy examples (where p_t is close to 1) and focuses training on hard, misclassified examples. With gamma = 2, an example with p_t = 0.9 contributes 100x less loss than with standard cross-entropy, while an example with p_t = 0.2 contributes a similar amount. This simple modification allowed RetinaNet to match the accuracy of two-stage detectors while maintaining the speed of one-stage detectors.

Core Concepts: Anchor Boxes, NMS, and IoU

Anchor Boxes

Anchor boxes are pre-defined bounding boxes of various scales and aspect ratios that serve as reference templates for detection. At each spatial position on the feature map, k anchor boxes are placed. For each anchor, the network predicts: the probability that it contains an object of each class, and 4 offsets to refine the anchor box to better fit the object.

The choice of anchor scales and ratios significantly affects detection performance. Standard choices: scales of {32, 64, 128, 256, 512} pixels (relative to the input image) and aspect ratios of {1:1, 1:2, 2:1}. These hyperparameters are typically set based on the dataset's object size distribution.

def generate_anchor_boxes(fmap_h, fmap_w, scales, aspect_ratios):
    """
    Generate anchor boxes for SSD / Faster R-CNN.
    Each feature map cell produces anchors at multiple scales and ratios.
    """
    anchors = []
    for i in range(fmap_h):
        for j in range(fmap_w):
            cx = (j + 0.5) / fmap_w  # Center x (normalized)
            cy = (i + 0.5) / fmap_h  # Center y (normalized)
            for scale in scales:
                for ratio in aspect_ratios:
                    w = scale * np.sqrt(ratio)
                    h = scale / np.sqrt(ratio)
                    anchors.append([cx, cy, w, h])
    return np.array(anchors)

Non-Maximum Suppression (NMS)

Detection networks typically produce multiple overlapping detections for the same object. NMS is the post-processing step that removes duplicates. It sorts detections by confidence score, selects the highest-scoring one, removes all other detections with IoU above a threshold (typically 0.5), and repeats for the remaining detections.

def non_max_suppression(boxes, scores, iou_threshold=0.5):
    """
    Non-maximum suppression to remove duplicate detections.
    boxes: (N, 4) array of [x1, y1, x2, y2] boxes
    scores: (N,) array of confidence scores
    """
    order = scores.argsort()[::-1]
    keep = []
    
    while len(order) > 0:
        i = order[0]
        keep.append(i)
        
        if len(order) == 1:
            break
        
        ious = compute_iou_batch(boxes[i], boxes[order[1:]])
        mask = ious <= iou_threshold
        order = order[1:][mask]
    
    return keep

Intersection over Union (IoU)

IoU measures the overlap between a predicted bounding box and a ground truth box: area of intersection divided by area of union. IoU > 0.5 is conventionally considered a true positive detection. More sophisticated metrics like GIoU, DIoU, and CIoU incorporate additional geometric constraints (distance between centres, aspect ratio consistency) and are used as loss functions in modern detectors.

Modern Detectors: DETR and Beyond

DETR (Detection Transformer, Carion et al., 2020) replaced the entire pipeline of anchor boxes, RPN, and NMS with a Transformer encoder-decoder architecture. It treats detection as a set prediction problem: the model outputs a fixed-size set of predictions, and bipartite matching (Hungarian algorithm) assigns each prediction to a ground truth box. DETR simplified the detection pipeline but was slow to converge and struggled with small objects.

Deformable DETR (Zhu et al., 2021) addressed these limitations by using multi-scale deformable attention, which attends to a small set of key sampling points around a reference point. This improved convergence speed and small object detection. Deformable DETR achieved state-of-the-art results on COCO while being faster and simpler than previous approaches.

Despite these advances, CNN-based detectors (particularly YOLOv8 and its successors) remain the most popular choice for production systems due to their well-understood behaviour, excellent tooling, and efficient deployment on edge devices.