Convolutional Neural Networks

Chapter Overview

This chapter focuses on CNNs as the image-oriented extension of the basic neural-network toolbox. The lecture covers convolution and pooling layers, preprocessing, the specific role of weight sharing, CNN backpropagation, example networks, and a gallery of major architectures from LeNet to ResNet.

It matters because the course uses CNNs as the canonical answer to image classification. The lecture repeatedly contrasts them with fully connected MLPs to explain why architectural bias matters.

The main idea is structural efficiency: instead of learning a different weight for every pixel-to-neuron connection, a CNN learns small filters reused across the image, thereby encoding locality and approximate translation invariance.

Important Definitions

• Convolution layer. Layer that applies learnable filters or kernels across spatial positions instead of using dense all-to-all connections.
• Kernel / filter. Small learnable tensor slid across the input to produce one feature map.
• Feature map. Activation map produced by one filter.
• Stride. Number of pixels by which the filter is shifted horizontally and vertically.
• Zero-padding. Border extension by zeros to control output size.
• Pooling layer. Layer that summarizes local neighborhoods independently in each depth slice and reduces spatial resolution.
• Max-pooling. Pooling operation keeping the maximum value in each pooling window.
• Weight sharing. Reusing the same kernel coefficients at every spatial location.
• Sparse connectivity. Each output neuron sees only a local receptive field rather than the entire input.
• Flatten layer. Reshape operation converting feature maps into a vector before dense layers; it has no trainable parameters.

Key Concepts and Intuition

• Why not a plain MLP for images? The lecture's answer is that dense networks ignore the spatial structure of the image and cannot efficiently detect the same pattern at different locations without exploding the number of parameters.
• Convolution uses the right inductive bias. The same small transformation is applied everywhere, which is appropriate when meaningful features such as edges or motifs can appear anywhere in the image.
• Pooling trades exact position for robust presence detection. If a small translation happens in the input, max-pooling often leaves the pooled representation almost unchanged.
• CNN backpropagation is conceptually the same as MLP backpropagation. The main change is that one parameter influences many spatial positions because of weight sharing.
• Architecture history is presented as a sequence of design ideas. LeNet introduces the classical stack, AlexNet scales it up with ReLU and GPU training, ZFNet uses visualization to improve the design, GoogLeNet reduces parameters with 1x1 convolutions, and ResNet fixes optimization depth issues with shortcut connections.

Mathematical Content

Convolution formulas.

The lecture stresses that kernels are small in width and height but extend through the full input depth. This is the key difference between spatial locality and channel completeness.

Main hyperparameters of a convolutional layer.

• Kernel size, typically 3x3 or 5x5.
• Number of filters / output depth.
• Stride, usually 1 and occasionally 2.
• Zero-padding, often chosen so input and output widths match.

Pooling.

The lecture uses spatial extent F and stride S, usually F = S = 2. Pooling operates independently on each feature map and has no trainable parameters.

Why convolution beats dense layers on images.

• Same transformation everywhere -> translation-aware feature detection.
• Far fewer parameters -> regularization, lower memory cost, better statistical efficiency, faster computation.
• Sparse local connections instead of dense global ones.

Typical CNN pipeline. Convolution -> activation -> pooling -> repeated feature extraction -> flatten -> fully connected classifier.

Data processing.

The lecture recommends channel-wise centering and normalization, plus data augmentation through sampling, translation, mirroring, rotation, and photometric transformations. Noise injection is also discussed as a regularization device.

Architectures explicitly covered.

• LeNet. Early conv-pool stack with average pooling and carefully structured partial connections in deeper conv layers.
• AlexNet. 11x11 first-layer filters with stride 4, ReLU, local response normalization, overlapping pooling, dropout, data augmentation, and a split across two GPUs.
• ZFNet. Uses deconvnet-style visualization to understand feature maps; improves AlexNet by reducing first-layer stride and filter size.
• VGGNet. Depth increase via many stacked small convolutions, illustrated in the VGG-16 diagram.
• GoogLeNet. Uses 1x1 reductions and inception modules to grow depth/width while keeping parameter count controlled; auxiliary classifiers support gradient flow during training.
• ResNet. Adds shortcut connections to make very deep optimization easier without extra parameters.

Illustrative comparison from the lecture lab slide.

A logistic network improves to a ReLU MLP, which improves again to a simple convolutional network on MNIST. This is used to justify the architecture, not just to report scores.

Distinctions and Comparisons

• Dense vs convolution. Dense layers have all-to-all, location-specific weights; convolution layers are sparse and share weights across space.
• Convolution vs pooling. Convolution has trainable filters and mixes information across depth; pooling has no trainable parameters and acts independently on each feature map.
• Average pooling vs max-pooling. Average pooling is historically common; max-pooling is presented as more common in practice and better for presence detection.
• Flatten vs learned layer. Flatten only reshapes the tensor; it does not change values and introduces no parameters.
• Architecture signatures. AlexNet -> two GPUs and ReLU-era scaling; ZFNet -> deconvnet-driven inspection; GoogLeNet -> 1x1 reduction and inception; ResNet -> shortcut connections.

Exam-Oriented Understanding

• When justifying CNN over MLP for images, mention both spatial structure and parameter efficiency.
• For parameter-count comparisons, the lecture's ranking is conceptually clear: dense layers have the most parameters, convolution fewer because of sparsity and sharing, pooling none.
• For convolution versus pooling, specify that convolution filters span the full input depth while pooling acts on each feature map separately.
• For flatten, say "reshape only" and "no trainable parameters".
• For architecture-matching questions, connect each name to the design idea rather than trying to memorize disconnected trivia.

Source References

• Deep Learning Lecture 3 (2026_DL_3.pdf), slides 7-31: Convolution operator, hyperparameters, zero-padding, sparse connectivity, pooling, canonical CNN pipeline.
• Deep Learning Lecture 3 (2026_DL_3.pdf), slides 33-47: Data preprocessing, data augmentation, noise-based regularization, CNN backpropagation, MNIST example networks.
• Deep Learning Lecture 3 (2026_DL_3.pdf), slides 56-120: Datasets and architecture gallery: LeNet, AlexNet, ZFNet, VGGNet, GoogLeNet, ResNet.

Relevant Exam Alignment

The sample exam asks exactly the kinds of questions this chapter prepares for: why CNNs fit image classification better than MLPs, dense-versus-conv-versus-pooling parameter comparisons, convolution-versus-pooling distinctions, flatten-layer role, and signature features of famous CNN families.

One important fidelity note: the provided lecture PDFs clearly cover LeNet, AlexNet, ZFNet, VGGNet, GoogLeNet, and ResNet. They do not explicitly cover Xception, YOLO, or segmentation/object-detection pipelines in the available material, so this synthesis stays within what is actually present in the lectures.